March 10, 2016
Case Study

Collaborative A/B Testing: Consumer Reports increases revenue per donation 32%

SUMMARY: At MarketingSherpa Summit 2016, almost a thousand marketers came together to learn in such tracks as Email & Mobile, Digital & Data and Content & Social. Read on to learn about the results of the live test with Consumer Reports.
by Paul Cheney, Senior Partnership Content Manager

Consumer Reports is the most well-known consumer advocacy organization in the U.S. Enlisting the help of the MarketingSherpa Summit audience, the team at Consumer Reports increased revenue per donation by 32%.


"This may actually be the largest collaborative A/B test on the planet," Austin McCraw, Senior Director of Content Production, said from the stage.

He was speaking to a crowd of around 1200 marketers at the Bellagio in Las Vegas during MarketingSherpa Summit 2016 in February.

The test he was referring to was an email test for Consumer Reports.

Many of us are familiar with the Consumer Reports' magazine and website, but some might not be as familiar with the fact that Consumer Reports is a nonprofit.

Consumer Reports formed as an independent nonprofit organization in 1936. It serves consumers through unbiased product testing and ratings, research, journalism, public education and advocacy.
  • Thousands of products are tested each year in its 50 state-of-the-art labs and 327-acre automotive testing track.

  • It engages with more than 1 million online activists to pass consumer protection laws in states and in Congress. It also holds corporations that do wrong by customers accountable, and encourages companies that are heading in the right direction.

  • Surveys of more than 1 million subscribers capture vital feedback on purchasing decisions, experiences and shopping habits, helping us further our work to inform and protect.

Joining McCraw on the stage to give more context to the audience was Dawn Nelson, Director of Fundraising, and Bruce Duesterhoeft, Program Manager of Online Fundraising, both from Consumer Reports.

"… Because we don't accept advertising," Nelson said to the crowd, "and … we are purchasing all the products we buy — it's [a] pretty expensive business ... while we get a fair amount of revenue from the subscriptions … it's not quite enough to cover all our costs, so to bridge that gap — which is about a 10% gap — we rely on our subscribers to give us an additional, optional donation."

Those donations are heavily reliant on Consumer Reports' membership program which is, in turn, heavily reliant on email as a marketing channel.

The email that would be sent for the test at Summit was the Control.

View the Creative Sample

Click here to see the full version of this creative sample

This email was to be sent to approximately 450,000 people. The average historic clickthrough rate is .54%.

The challenge was to increase that clickthrough rate and, ultimately, to achieve more revenue from donations.

"Pick a winner," Duesterhoeft said to the audience.


Before the test from the stage at the Summit, the team ran a pre-test to gather some insight on the Consumer Reports donor base.

Consumer Reports had a prior email that they tested with the same messaging as the Control for the Summit test.

"It's very similar in terms of the approach and the design of the email," McCraw said.

The Pre-Test: Is the value proposition too unfocused?

View the Creative Sample

Click here to see the full version of this creative sample

When the team analyzed it, they realized that there was a mixed focus regarding the value proposition.

In this first test, the team hypothesized that the value proposition of the Control email didn't focus enough on one aspect of the value proposition. With that in mind, the team created three potential value focuses. Each value proposition focus answered a critical question in the team's mind:

What was the best reason to give to Consumer Reports?
  • Value Proposition Focus #1: Because of Consumer Reports' honest and unbiased reporting?

  • Value Proposition Focus #2: Because of the personal impact Consumer Reports is having?

  • Value Proposition Focus #3: Because of the unparalleled quality of Consumer Reports' research?

"We were really honing in on the value proposition," McCraw said, describing the categories for the test.

Approach: Crowdsourcing the email copy

To answer these questions, the team worked with MarketingExperiments and Convince & Convert blog audiences.

View the Creative Sample

Click here to see the full version of this creative sample

They received over 54 submissions. Each of the submissions fell into one of the value categories determined by the team. After doing some statistical modelling, however, the team realized they could run a fourth treatment.

The final Treatment's value category was chosen by the audience as well.

View the Creative Sample

Click here to see the full version of this creative sample

With the final treatments done, the team ran the test.

View the Creative Sample

Click here to see the full version of this creative sample

Results: The crowd suffers a setback

It turned out that the crowd didn't win — in any of the treatments. "Every single one of the treatments," McCraw said, "underperformed the Control."

View the Creative Sample

Click here to see the full version of this creative sample

Hypothesis: Three potential reasons why the treatments lost

The discovery from the test was even more interesting than the results. The team hypothesized that there were a few potential reasons for the loss.
  • Reason #1: Perhaps focusing on just one element of the value proposition alienates certain groups of donors.

  • Reason #2: Perhaps there is some unnecessary friction in the layout/structure of the copy.

  • Reason #3: Perhaps there is confusion around what the actual member benefits are and potential donors are most motivated by learning about the specific member benefits.

The Summit Test: Are membership benefits too de-emphasized?

Taking those potential reasons, the team again converted them into options for the next test's treatments.
  1. Emphasizing (not Isolating) Value: Treatments will feature copy that emphasizes (not isolates) a particular element of the value proposition.

  2. Reducing Friction: Treatments will feature copy that is simplified, shorter and easier to digest.

  3. Explaining Member Benefits/Incentives: Treatments will feature copy that brings greater clarity to the actual benefits of becoming a donor/member.

Approach: Crowdsourcing the email copy again

This time, with the Summit audience rather than the blog audience, the team gave the crowd another chance. The first task was for the audience to choose the test category from those above.

Put to a live survey, the audience chose the third option (explaining member benefits/incentives).

View the Creative Sample

Click here to see the full version of this creative sample

The next task was to choose the actual treatments. This time, because of the smaller changes, the team could only run two treatments against the Control.

The Summit audience was presented with four treatments to choose from:

View the Creative Sample

Click here to see the full version of this creative sample

McCraw gave four quick summaries of each of the treatments:
  • Treatment 1: "All we're doing in Treatment 1 is adding a paragraph that introduces and talks about those member benefits."

  • Treatment 2: "All we're going to do is move that membership benefits link up into the body of the email."

  • Treatment 3: "We're going to have a call-out-box that describes the benefits that they're going to get."

  • Treatment 4: "Now we're really fleshing out not just what you get but the tiers of membership and what comes along with those different tiers."

By polling the audience again, the treatments were chosen:

View the Creative Sample

Click here to see the full version of this creative sample

Treatments 3 and 4 won. And, with that, the test was ready to be sent to the Consumer Reports list. But in order to get the results, the Summit audience needed to wait until the next morning.

View the Creative Sample

Click here to see the full version of this creative sample

The Final Results: The crowd (ultimately) prevails

The next day, the results of the test were in.

"I have some good news and some bad news," McCraw said.

At first glance the treatments lost — again.

View the Creative Sample

Click here to see the full version of this creative sample

But, upon closer examination, the treatments increased revenue per donation by 32%, with total revenue increasing by 16%.

View the Creative Sample

Click here to see the full version of this creative sample

Discovery: Benefits work. Long copy, maybe not

There are two potential reasons why this result may have happened. All of them are just more hypotheses to be tested.

  • Both underperforming treatments increased the length and complexity of the email (friction) by including more information. Perhaps we will see a greater increase in response as we reduce the length and complexity of the email.

  • Treatment 2, which was the only treatment that revealed the $100 membership level, produced the highest overall revenue per donation. Perhaps by emphasizing the higher membership level, we will motivate prospects to donate more per donation.

Nelson believed the Summit audience did a wonderful job in choosing the treatments

"It was really fun for us," Nelson said, referring to waiting on the results, "… I was like a kid on Christmas morning."

Related Resources

Online Testing: 3 test options, 3 possible discoveries, 1 live test from Email Summit 2014 [From the MarketingExperiments blog]

Live From MarketingSherpa Summit 2016: Humana on the power of iterative testing [From the MarketingSherpa blog]

Live Experiment (Part 2): Real testing is messy [From the MarketingExperiments blog]

Email Summit 2015 Live Optimization [From MECLABS Institute]

A/B Testing: How a landing page test yielded a 6% increase in leads

Landing Page Optimization: Simple, value-infused page increases leads 8% in 24-hour test

Improve Your Marketing

Join our thousands of weekly case study readers.

Enter your email below to receive MarketingSherpa news, updates, and promotions:

Note: Already a subscriber? Want to add a subscription?
Click Here to Manage Subscriptions