“In effect, BuzzFeed’s devotion to rapid A/B testing was also an acknowledgement of sorts: ‘We don’t know what works.’” – Bharat Anand in The Content Trap
Does your marketing department embrace that same humility?
To inspire your search for what works with your brand, read on for examples from Codecademy, an energy shopping website, and a federal credit union.
This article was originally published in the MarketingSherpa email newsletter.
“Any piece of marketing data that doesn’t help you predict behavior is virtually useless to you as a marketer,” said Flint McGlaughlin in Unlock the Power of Your A/B Testing Program.
In this article, we bring you stories of companies that ran experiments to create data that would help them predict behavior of not only customers, but also fellow website owners.
First up, a look at how Codecademy discovered the most effective way to present its pricing to potential students. Then, an energy shopping website that ran landing page tests to predict how customers would react to a simpler signup process. And finally, a credit union that ran an SEO experiment across six different strategies to predict the best way to build more links to its website.
As a subscription business, Codecademy knew that small repeatable wins could compound to earn them millions of dollars in additional revenue.
"When I think back to where we were two and a half years ago, the biggest issue with our testing program was that we tried several small ideas in different places and didn’t iterate enough on concepts. Our ideas were driven by individual opinions and changes that we saw our competitors making, which is one of the worst ways to run a testing program,” said Daniel Layfield, Product Manager, Codecademy.
“The tide turned for us when we began focusing on our own learners and what we knew about them—their perceptions of our product and consumer behavior that was supported by evidence. This research informed better hypotheses, which allowed us to launch better tests and start unraveling bigger problems—problems we couldn't solve without changing fundamental parts of our business. That’s what led us to strategic testing. Once we realized this, we had to prioritize which areas of the business to test first,” he said.
One of the first strategic tests the growth team prioritized was to A/B test reordering the pricing plan cards, with the hypothesis that shifting the order to start with the lowest price instead of the shortest plan length would increase the value perception of the higher LTV (lifetime value) plans.
The original (control) was ordered from monthly to annual plans and included the percentage customers could save by signing up for a longer plan.
Creative Sample #1: Control (original) price presentation for Codecademy
The after (treatment) is ordered from the lowest monthly cost to the highest.
It also leveraged a psychological principle called the Rule of 100, which suggests that users perceive dollar amounts over 100 as being greater in value than percentages, even if both equate to the same amount. Because the savings on the annual plan was above $100, the team tested showing dollar amounts.
Creative Sample #2: Treatment (after) price presentation for Codecademy
The “after” (treatment) test variant contributed to a 28% lift in annual pro plans, increasing the average order value as well as the lifetime value of new customers.
“Use research, persuasion principles, and experimentation to find the best pricing presentation that works for your users and achieves business goals,” advised Ben Labay, Managing Director, Speero (Codecademy’s customer experience agency).
Power Target LLC runs energy shopping websites like ElectricityRates.com to help consumers in deregulated energy markets make more informed, confident choices while providing a simple online marketplace for switching electricity providers.
The team ran a landing page test to better understand customers and increase sales. For Power Target, a sale is a customer enrolling in a new electricity plan.
Users coming from paid traffic were landing on a generic experience with a little bit of information about what the site offered and a ZIP code entry form visitors had to fill out to take the next step and compare energy rates in their area. This experience was not specific to any market or customer segment and while it was effective for high-intent visitors, a lot of users were bouncing.
Creative Sample #3: Before (control) paid traffic landing page for energy shopping website
The team created a new (treatment) landing page that has a relevant headline and displays energy options from the user’s utility before they have to enter any information. The team’s hypothesis was that by displaying energy options and cutting out the step of having to enter the ZIP code first, users would essentially be able to instantly “window-shop” by seeing what’s available in their area immediately upon landing.
Creative Sample #4: Treatment (after) paid traffic landing page for energy shopping website
The A/B split test ran for about four weeks and the treatment improved conversion rates by 32% over the previous version and was rolled out to all paid traffic coming to ElectricityRates.com.
“Give your customer what they're looking for as soon as they land on your site. Eliminate any unnecessary steps to make the buying process as smooth as possible and constantly question ‘do we really need this step/process in order to better serve the customer?’” advised Adam Cain, Director of Customer Experience, Power Target LLC. “Also make your landing experience as relevant as possible to the user’s intent including their location, demographics, etc.”
Over a seven-month span of link building, the team at Truliant Federal Credit Union secured a total of 77 backlinks for the financial institution’s website by testing six different strategies. Of the 77 links acquired, over 70% were to target pages. Here are the strategies they experimented with:
“In addition to ranking keyword and traffic growth, we also saw improvements on the Page Authority (PA) score from Moz on the same pages,” Hibbard said.
Get Better Business Results With a Skillfully Applied Customer-first Marketing Strategy
The customer-first approach of MarketingSherpa’s agency services can help you build the most effective strategy to serve customers and improve results, and then implement it across every customer touchpoint.Get More Info >
Project and Ideas Pitch Template
A free template to help you win approval for your proposed projects and campaignsGet the Template >
Six Quick CTA checklists
These CTA checklists are specifically designed for your team — something practical to hold up against your CTAs to help the time-pressed marketer quickly consider the customer psychology of your “asks” and how you can improve them.Get the Checklists >
Infographic: How to Create a Model of Your Customer’s Mind
You need a repeatable methodology focused on building your organization’s customer wisdom throughout your campaigns and websites. This infographic can get you started.Get the Infographic >
Infographic: 21 Psychological Elements that Power Effective Web Design
To build an effective page from scratch, you need to begin with the psychology of your customer. This infographic can get you started.Get the Infographic >
7 Steps to Discovering Your Essential Value Proposition with Simple A/B Tests
Drive better results when you discover what it is about your business that customers love.Get the Guide >
Receive the latest case studies and data on email, lead gen, and social media along with MarketingSherpa updates and promotions.