We look back at Obama’s years as President… of split-testing

Obama Split Testing

We look back at Obama’s years as President… of split-testing

Throughout his two elections, conversion rate optimisation was fundamental to Obama’s success.

In 2008, he pioneered the use of a/b testing for boosting political campaign performance. Following on from this success, in 2012 Obama sourced expertise from Kyle Rush, the former Head of Optimisation at Optimizely. Assisted by his excellent digital team he ran around 500 tests, resulting in a 49% uplift in donations & raising $690 million in funds.

 

barack-obama

500 A/B tests

49% donations uplift

$690 million raised 

 

 

So how did he do it?

  1. He knew what to test

  2. He knew how to test

  3. He Invested. And it was worth it.

1. He knew what to test

 

Split testing takes the guess work out of optimisation. It gives us the ability to find out the truth about our users and what makes them tick. However, the question of deciding what to test still falls victim to subjectivity and wrong assumptions. This often results in fruitless ideas & weak hypotheses.

How did Obama combat this pitfall? His test hypotheses were grounded by a combination of intelligent analytics and user psychology.

 

Intelligent analytics to develop hypotheses

One of the vital elements needed to develop strong test hypotheses is skilful analysis to gain actionable insights. It’s vital to delve into qualitative and quantitative data to really understand your users and to uncover potential pages, elements, segments, user journeys & experiences for optimisation. You need to gain insight into areas with the highest potential uplifts and ROI. You need expertise in conducting thorough analysis efficiently – pulling out the valuable insights to develop well-formed hypotheses. The ideal optimisation strategy adopts a holistic approach, gaining this insight through a variety of methods: data analytics, user testing, heat maps, session replay, surveys and so on.

In 2012, Obama’s skilled optimisation team conducted thorough usability testing and identified problem areas of the onsite donation form that were causing friction. They hypothesised that restructuring the form into a multi-step ‘sequential’ form rather than one long page would give the impression of a quicker process and present the user with fewer psychological frictions at once.

They also used analysis of validation errors to develop an alternative field order: putting the easiest fields first. This was done with the insight that having invested time in filling the first few easier fields, when the user reaches the trickier fields they will be more likely to persist and complete the rest of the form.

barack obama form optimisation split test

The team’s hypothesis was proved through an a/b test: the multi-step sequential process achieved a 5% uplift in donations compared to the original version. And given the amount of traffic that Obama’s form was receiving, the revenue gains were vast.

Testing is an iterative process of continual learning. Obama’s team ran tests on the new, sequential donation form to further optimise the conversion rate and extract maximum donations possible from visitors.

 

User Psychology to develop hypotheses

Without the application of neuroscience and behavioural economics businesses find themselves continually concluding tests without revenue uplifts. Conversely, with the right knowledge of psychological principles within conversion optimisation, you can expect big uplifts. This isn’t to say that with these insights you can expect every test to be a winner – otherwise why would we test at all? – but it means you will be able to achieve infinitely larger revenue gains and an extraordinary ROI.

(iv) One psychological principle Obama’s team applied during testing was the Information Gap Theory of Curiosity, (Loewenstein, 1994). Curiosity is our response to the information gap – the gap between what we know and what we could find out. Curiosity is an emotion; an inexplicable yearning to find out abstract information. Our strongest feelings of curiosity arise when we know just a little about the subject. When our curiosity is provoked, a part of the brain known as the caudate is activated. The caudate has been found to sit at the intersection of new knowledge and positive emotions. Our curiosity begins as a dopaminergic craving, like our other primal urges, for food and sex. Using this principle Obama’s team developed a strong hypothesis: By piquing users’ curiosity with carefully crafted email subject lines that play on the information gap, open rates will be boosted, ultimately leading to an uplift in donations.

 

Some of the email subject lines tested during Obama’s campaign:

subject line split test
Image courtesy of NY Mag

 

 

Not only did the team develop their subject line hypothesis based on the psychological principle of the information gap but they also grounded the hypothesis with thorough research into their audience.

One of the reasons for Obama’s extraordinary success is that, where other politicians fail, he knew his audience was seeking someone likable & personable. He created relationships with the public via a range of media so they would emotionally invest in his quest for presidency. He extended this approach to all areas of his campaign, even down to his email subject lines. He made them personal and informal. He was mocked for his subject lines’ resemblance to spam but this was not a concern for the team because we all know exactly who Obama is. Not every business could get away with such a strategy – these lines could produce disastrous open rates for some, which highlights the importance of knowing your audience and testing hypotheses.

Furthermore, Obama stayed savvy to the diminishing effects of certain approaches: he knew users would become accustomed to certain types of ‘curiosity piquing’ lines, meaning the information gap would decline and become less affective. The team continually tested new copy to overcome this.

 

Segmentation & Personalisation to develop hypotheses

Segmentation can be seen as a branch of either analytics or user psychology to develop hypotheses, depending on the way it is applied. Advanced testing strategies will gain deeper analytics insights by segmenting their audiences and avoiding the assumption of homogenous behaviour across the total population. They will also exploit the benefits of personalisation in terms of user psychology.

In October 2016 mobile and tablet web browsing overtook desktop for the first time, with a 51% share of total global online browsing. The majority of the population is glued to their mobile, and taking advantage of this is key to online success.

Obama’s team had a strong mobile strategy, connecting with people through text alerts and ensuring donations could be made directly via mobile. Furthermore, they developed a mobile optimised site and they knew that what might earn an increase in donations on desktop may do the opposite on mobile, so they segmented data accordingly.

Obama’s team were also careful not to make assumptions about their mobile users and what they will and won’t respond to. In mobile specific optimisation, it’s standard to simplify and remove unnecessary elements due to the reduced screen resolution. However, Obama’s team were careful not to assume which elements were ‘unnecessary’ for the mobile experience.

For example, in a promotion to encourage donations by offering the chance to go for dinner with Obama, using imagery of the man himself proved highly effective in the desktop experience. The team hypothesised that adding this imagery on the mobile experience would lead to an increase in donations – despite the fact that it would push the donation form down the page due to the limited screen real estate compared to desktop. The team were right and adding the imagery achieved a 6.9% uplift in donations.

 

mobile split test obama

 

 

Obama’s team used segmentation and personalisation to their full potential. They served personalised landing pages dependent on the campaign from which the user arrived:

Reddit split test obama

Image courtesy of Forbes

 

 

They also segmented data by location and personalised content accordingly:

location personalisation obama split test
Personalised sign up form dependent on user’s location to increase relevance, engagement and build rapport

 

Moreover, they personalised additional donation requests dependent on the user’s initial donation amount to maximise average donation value.

A clever strategy will segment by device in addition to a multitude of other dimensions: behaviour, location, technology, traffic source, campaign and a variety of other custom attributes.

Furthermore, they’ll perform both pre-test segmentation (as Obama did in the examples above) and post-test segmentation (For example, there may be an audience segment X who prefer version A, despite the overall population favouring version B. Hence serving version A to segment X and version B to the remaining population will maximise conversion)

 

2. He knew how to test

 

A team that knows how to test is imperative for testing success.

Tick Point Obama’s team knew how to implement tests efficiently. They had a fast run-rate to achieve maximum gains. They ran approximately 500 tests in the lead up to the 2012 election, and ensured there was no interference skewing the data when running multiple tests at once.

Tick Point They selected their platform (Optimizely) wisely to get the most out of their tests and maximum ROI. Moreover, they knew how to use this platform to its full potential.

Tick Point They implemented intelligent analytics to measure the success of each test, with custom metrics to get insights beyond conversion and revenue, assisting their iterative testing process. They applied segmentation and personalisation intelligently.

Tick Point They understood the statistics behind testing. They knew how to interpret the data. They knew the considerations of statistical significance in testing.

Tick Point They ensured the changes they applied were in line with Obama’s wider campaign strategy to give the audience a seamless and consistent experience across media.

Tick Point They prioritised campaigns sensibly to maximise revenue.

Tick Point They were organised into a team structure which facilitated optimisation of the site in an efficient and effective way.

 

3. He invested in his team and his strategy. And it was worth it

Obama set the trend for applying conversion rate optimisation & testing to political campaigns back in 2008 with future candidates and presidents to follow in his steps. He could foresee the remarkable value he would achieve by investing in a highly skilled team.

You can see the influence Obama had on Hillary Clinton and Donald Trump who both adopted the sequential donation form style in their own campaigns running up to the election in 2016:

trump copies obamas split testing

 

hillary copies obamas split testing

 

Besides a monetary investment, the team invested their time in fostering a strong testing culture. This meant circulating results and engaging all departments; encouraging all marketers to question their assumptions; implementing testing as an iterative process of continual learning.

As you know, Obama’s investment in testing paid off. This is because he had a team of skilled conversion optimisers with a wealth of knowledge in testing, data analysis and the application of psychology to produce vast revenue gains.

You could have the same – check our conversion optimisation services and get in touch with the Neuro Web Marketing team to get started today.

obama out

Written by Saskia Cook

Saskia is a Split Testing Specialist at Neuro Web Marketing, with a background in economics and statistical analysis. She is always passionate to learn the latest in CRO and apply her experience in testing and optimisation across a diverse range of sites and sectors

Leave a Reply

Your email address will not be published. Required fields are marked *

Subscription

Free subscription to more than £5million worth of research, optimisation and testing and insights from 13 years of web psychology.

Agency services

Barely have enough time to read this blog? Let alone to apply this work to your site? You're in luck - we do the following every day:

  • Site-wide conversion audits
  • Optimising your checkout processes
  • Crafting Value Propositions
  • Optimising Landing Pages
  • Improving SEO & Google Rankings
  • Link detoxification
  • Penalisation recovery
Get in touch