We look back at Obama’s years as President… of split-testing
We look back at Obama’s years as President… of split-testing
Throughout his two elections, conversion rate optimisation was fundamental to Obama’s success.
In 2008, he pioneered the use of a/b testing for boosting political campaign performance. Following on from this success, in 2012 Obama sourced expertise from Kyle Rush, the former Head of Optimisation at Optimizely. Assisted by his excellent digital team he ran around 500 tests, resulting in a 49% uplift in donations & raising $690 million in funds.
500 A/B tests
49% donations uplift
$690 million raised
So how did he do it?
He knew what to test
He knew how to test
He Invested. And it was worth it.
1. He knew what to test
Split testing takes the guess work out of optimisation. It gives us the ability to find out the truth about our users and what makes them tick. However, the question of deciding what to test still falls victim to subjectivity and wrong assumptions. This often results in fruitless ideas & weak hypotheses.
How did Obama combat this pitfall? His test hypotheses were grounded by a combination of intelligent analytics and user psychology.
Intelligent analytics to develop hypotheses
One of the vital elements needed to develop strong test hypotheses is skilful analysis to gain actionable insights. It’s vital to delve into qualitative and quantitative data to really understand your users and to uncover potential pages, elements, segments, user journeys & experiences for optimisation. You need to gain insight into areas with the highest potential uplifts and ROI. You need expertise in conducting thorough analysis efficiently – pulling out the valuable insights to develop well-formed hypotheses. The ideal optimisation strategy adopts a holistic approach, gaining this insight through a variety of methods: data analytics, user testing, heat maps, session replay, surveys and so on.
In 2012, Obama’s skilled optimisation team conducted thorough usability testing and identified problem areas of the onsite donation form that were causing friction. They hypothesised that restructuring the form into a multi-step ‘sequential’ form rather than one long page would give the impression of a quicker process and present the user with fewer psychological frictions at once.
They also used analysis of validation errors to develop an alternative field order: putting the easiest fields first. This was done with the insight that having invested time in filling the first few easier fields, when the user reaches the trickier fields they will be more likely to persist and complete the rest of the form.
The team’s hypothesis was proved through an a/b test: the multi-step sequential process achieved a 5% uplift in donations compared to the original version. And given the amount of traffic that Obama’s form was receiving, the revenue gains were vast.
Testing is an iterative process of continual learning. Obama’s team ran tests on the new, sequential donation form to further optimise the conversion rate and extract maximum donations possible from visitors.
User Psychology to develop hypotheses
Without the application of neuroscience and behavioural economics businesses find themselves continually concluding tests without revenue uplifts. Conversely, with the right knowledge of psychological principles within conversion optimisation, you can expect big uplifts. This isn’t to say that with these insights you can expect every test to be a winner – otherwise why would we test at all? – but it means you will be able to achieve infinitely larger revenue gains and an extraordinary ROI.
(iv) One psychological principle Obama’s team applied during testing was the Information Gap Theory of Curiosity, (Loewenstein, 1994). Curiosity is our response to the information gap – the gap between what we know and what we could find out. Curiosity is an emotion; an inexplicable yearning to find out abstract information. Our strongest feelings of curiosity arise when we know just a little about the subject. When our curiosity is provoked, a part of the brain known as the caudate is activated. The caudate has been found to sit at the intersection of new knowledge and positive emotions. Our curiosity begins as a dopaminergic craving, like our other primal urges, for food and sex. Using this principle Obama’s team developed a strong hypothesis: By piquing users’ curiosity with carefully crafted email subject lines that play on the information gap, open rates will be boosted, ultimately leading to an uplift in donations.
Some of the email subject lines tested during Obama’s campaign:
Image courtesy of NY Mag
Not only did the team develop their subject line hypothesis based on the psychological principle of the information gap but they also grounded the hypothesis with thorough research into their audience.
One of the reasons for Obama’s extraordinary success is that, where other politicians fail, he knew his audience was seeking someone likable & personable. He created relationships with the public via a range of media so they would emotionally invest in his quest for presidency. He extended this approach to all areas of his campaign, even down to his email subject lines. He made them personal and informal. He was mocked for his subject lines’ resemblance to spam but this was not a concern for the team because we all know exactly who Obama is. Not every business could get away with such a strategy – these lines could produce disastrous open rates for some, which highlights the importance of knowing your audience and testing hypotheses.
Furthermore, Obama stayed savvy to the diminishing effects of certain approaches: he knew users would become accustomed to certain types of ‘curiosity piquing’ lines, meaning the information gap would decline and become less affective. The team continually tested new copy to overcome this.
Segmentation & Personalisation to develop hypotheses
In October 2016 mobile and tablet web browsing overtook desktop for the first time, with a 51% share of total global online browsing. The majority of the population is glued to their mobile, and taking advantage of this is key to online success.
Obama’s team had a strong mobile strategy, connecting with people through text alerts and ensuring donations could be made directly via mobile. Furthermore, they developed a mobile optimised site and they knew that what might earn an increase in donations on desktop may do the opposite on mobile, so they segmented data accordingly.
Obama’s team were also careful not to make assumptions about their mobile users and what they will and won’t respond to. In mobile specific optimisation, it’s standard to simplify and remove unnecessary elements due to the reduced screen resolution. However, Obama’s team were careful not to assume which elements were ‘unnecessary’ for the mobile experience.
For example, in a promotion to encourage donations by offering the chance to go for dinner with Obama, using imagery of the man himself proved highly effective in the desktop experience. The team hypothesised that adding this imagery on the mobile experience would lead to an increase in donations – despite the fact that it would push the donation form down the page due to the limited screen real estate compared to desktop. The team were right and adding the imagery achieved a 6.9% uplift in donations.
Obama’s team used segmentation and personalisation to their full potential. They served personalised landing pages dependent on the campaign from which the user arrived:
Image courtesy of Forbes
They also segmented data by location and personalised content accordingly:
Moreover, they personalised additional donation requests dependent on the user’s initial donation amount to maximise average donation value.
A clever strategy will segment by device in addition to a multitude of other dimensions: behaviour, location, technology, traffic source, campaign and a variety of other custom attributes.
Furthermore, they’ll perform both pre-test segmentation (as Obama did in the examples above) and post-test segmentation (For example, there may be an audience segment X who prefer version A, despite the overall population favouring version B. Hence serving version A to segment X and version B to the remaining population will maximise conversion)
2. He knew how to test
A team that knows how to test is imperative for testing success.
Obama’s team knew how to implement tests efficiently. They had a fast run-rate to achieve maximum gains. They ran approximately 500 tests in the lead up to the 2012 election, and ensured there was no interference skewing the data when running multiple tests at once.
They selected their platform (Optimizely) wisely to get the most out of their tests and maximum ROI. Moreover, they knew how to use this platform to its full potential.
They implemented intelligent analytics to measure the success of each test, with custom metrics to get insights beyond conversion and revenue, assisting their iterative testing process. They applied segmentation and personalisation intelligently.
They understood the statistics behind testing. They knew how to interpret the data. They knew the considerations of statistical significance in testing.
They ensured the changes they applied were in line with Obama’s wider campaign strategy to give the audience a seamless and consistent experience across media.
They prioritised campaigns sensibly to maximise revenue.
They were organised into a team structure which facilitated optimisation of the site in an efficient and effective way.
3. He invested in his team and his strategy. And it was worth it
Obama set the trend for applying conversion rate optimisation & testing to political campaigns back in 2008 with future candidates and presidents to follow in his steps. He could foresee the remarkable value he would achieve by investing in a highly skilled team.
You can see the influence Obama had on Hillary Clinton and Donald Trump who both adopted the sequential donation form style in their own campaigns running up to the election in 2016:
Besides a monetary investment, the team invested their time in fostering a strong testing culture. This meant circulating results and engaging all departments; encouraging all marketers to question their assumptions; implementing testing as an iterative process of continual learning.
As you know, Obama’s investment in testing paid off. This is because he had a team of skilled conversion optimisers with a wealth of knowledge in testing, data analysis and the application of psychology to produce vast revenue gains.
You could have the same – check our conversion optimisation services and get in touch with the Neuro Web Marketing team to get started today.