Join thousands of weekly readers and receive practical marketing advice for FREE.
MarketingSherpa's Case Studies, New Research Data, How-tos, Interviews and Articles

Enter your email below to join thousands of marketers and get FREE weekly newsletters with practical Case Studies, research and training, as well as MarketingSherpa updates and promotions.


Please refer to our Privacy Policy and About Us page for contact details.

No thanks, take me to MarketingSherpa

First Name:
Last Name:
Oct 15, 2013

Marketing Research Chart: Does A/B testing deliver a positive ROI?

SUMMARY: Which call-to-action button works best? Should I use a long or short headline? What color should my CTA buttons be?

Much like asking "What is the best meal on your menu?" there is not one single correct answer for the above questions, there is only the answer that is best for your unique company with your unique customers.

To get that unique answer, marketers can use A/B testing to see what really works for their customers. In this week's chart, we explore the financial side of A/B testing.
by Daniel Burstein, Director of Editorial Content

While it would be valuable to really know the most effective marketing messaging, let's face it, A/B testing takes a lot of work. You need:
  • The IT resources and tools to create and measure a split test

  • To create two versions (at least) of every element you want to test

  • A testing methodology to ensure you run valid tests, and learn from those tests

Is all of this extra work really worth it? To find out, in the MarketingSherpa 2012 Website Optimization Benchmark Report, we asked marketers

Q: Did optimization or testing demonstrate ROI in 2011?

View Chart Online

Click here to see a printable version of this chart

Many marketers are not measuring the ROI of testing

The majority of marketers could not or did not calculate ROI for conversion rate optimization and A/B testing. Calculating that ROI can be difficult, and it is an imperfect science. You have to include not only, for example, a software platform purchase, but also the extra time a designer had to take to produce not only the original, but a treatment version as well.

Calculating the ROI can be extremely valuable. After all, demonstrating return on investment is the way marketers are able to secure the resources to continue to run the tactics that are effective for their companies (and, dare I say, the way marketers are able to earn promotions after proving their success to business leaders).

If you don't know, well now you know

If you are among the marketers who aren't measuring split testing ROI, here's the good news — it is very likely positive.

After all, 38% of marketers demonstrated a positive ROI with their testing efforts, for 6% of marketers testing was a wash, and only 3% experienced a negative ROI.

And, that ROI can be extremely powerful

$200 million in extra revenue

When I asked Amelia Showalter, Director of Digital Analytics, Obama for America, about the amount of campaign donations that could be directly attributed to improvements thanks to A/B testing, here's what she had to say.

"When we're working on the campaign, we're actually working so hard to run all those tests that we didn't always keep perfect track of exactly what results were long term. It's hard to calculate this stuff out when we want to put all our resources into running more tests. So, we don't actually ever have a perfect estimate of actually how much extra revenue was due to our testing, but I think that $200 million is a fairly reasonable estimate."

So, if you aren't calculating the revenue of your A/B testing you can take heart that, yes, it is quite hard, even for the advanced professionals.

You can also take heart that if you're running valid tests, you are likely improving the bottom line.

So, give it a try. While your first calculations won't be perfect, and you don't want to take too many resources from running the tests themselves, those ROI numbers can help you gain the resources you need to continue, and improve, your testing.

Related Resources

Email Optimization: A discussion about how A/B testing generated $500 million in donations

MECLABS Online Testing Course

Website Optimization: Simple A/B test adds 'thousands and thousands of dollars' to's bottom line

See Also:

Post a Comment

Note: Comments are lightly moderated. We post all comments without editing as long as they
(a) relate to the topic at hand,
(b) do not contain offensive content, and
(c) are not overt sales pitches for your company's own products/services.

To help us prevent spam, please type the numbers
(including dashes) you see in the image below.*

Invalid entry - please re-enter

*Please Note: Your comment will not appear immediately --
article comments are approved by a moderator.

Improve Your Marketing

Join our thousands of weekly Case Study readers.

Enter your email below to receive MarketingSherpa news, updates, and promotions:

Note: Already a subscriber? Want to add a subscription?
Click Here to Manage Subscriptions

Best of the Week:
Marketing case studies and research

Chart Of The Week

B2B Marketing

Consumer Marketing

Email marketing

Inbound Marketing

SherpaStore Alerts


We value your privacy and will not rent or sell your email address. Visit our About Us page for contact details.