Subscription Marketing Experiments: Quick case studies about pricing, landing page, and SEO tests for recurring revenue businesses

This article was originally published in the MarketingSherpa email newsletter.

“Any piece of marketing data that doesn’t help you predict behavior is virtually useless to you as a marketer,” said Flint McGlaughlin in Unlock the Power of Your A/B Testing Program.

In this article, we bring you stories of companies that ran experiments to create data that would help them predict behavior of not only customers, but also fellow website owners.

First up, a look at how Codecademy discovered the most effective way to present its pricing to potential students. Then, an energy shopping website that ran landing page tests to predict how customers would react to a simpler signup process. And finally, a credit union that ran an SEO experiment across six different strategies to predict the best way to build more links to its website.

Quick Case Study #1: Rule of 100 and ordering in pricing presentation contribute to a 28% lift in annual pro plans for Codecademy

As a subscription business, Codecademy knew that small repeatable wins could compound to earn them millions of dollars in additional revenue.

"When I think back to where we were two and a half years ago, the biggest issue with our testing program was that we tried several small ideas in different places and didn’t iterate enough on concepts. Our ideas were driven by individual opinions and changes that we saw our competitors making, which is one of the worst ways to run a testing program,” said Daniel Layfield, Product Manager, Codecademy.

“The tide turned for us when we began focusing on our own learners and what we knew about them—their perceptions of our product and consumer behavior that was supported by evidence. This research informed better hypotheses, which allowed us to launch better tests and start unraveling bigger problems—problems we couldn't solve without changing fundamental parts of our business. That’s what led us to strategic testing. Once we realized this, we had to prioritize which areas of the business to test first,” he said.

One of the first strategic tests the growth team prioritized was to A/B test reordering the pricing plan cards, with the hypothesis that shifting the order to start with the lowest price instead of the shortest plan length would increase the value perception of the higher LTV (lifetime value) plans.

The original (control) was ordered from monthly to annual plans and included the percentage customers could save by signing up for a longer plan.

Creative Sample #1: Control (original) price presentation for Codecademy

The after (treatment) is ordered from the lowest monthly cost to the highest. 

It also leveraged a psychological principle called the Rule of 100, which suggests that users perceive dollar amounts over 100 as being greater in value than percentages, even if both equate to the same amount. Because the savings on the annual plan was above $100, the team tested showing dollar amounts.

Creative Sample #2: Treatment (after) price presentation for Codecademy

The “after” (treatment) test variant contributed to a 28% lift in annual pro plans, increasing the average order value as well as the lifetime value of new customers.

“Use research, persuasion principles, and experimentation to find the best pricing presentation that works for your users and achieves business goals,” advised Ben Labay, Managing Director, Speero (Codecademy’s customer experience agency).

Quick Case Study #2: Reducing friction increases sales 32% for energy shopping website

Power Target LLC runs energy shopping websites like ElectricityRates.com to help consumers in deregulated energy markets make more informed, confident choices while providing a simple online marketplace for switching electricity providers.

The team ran a landing page test to better understand customers and increase sales. For Power Target, a sale is a customer enrolling in a new electricity plan.

Users coming from paid traffic were landing on a generic experience with a little bit of information about what the site offered and a ZIP code entry form visitors had to fill out to take the next step and compare energy rates in their area. This experience was not specific to any market or customer segment and while it was effective for high-intent visitors, a lot of users were bouncing.

Creative Sample #3: Before (control) paid traffic landing page for energy shopping website

The team created a new (treatment) landing page that has a relevant headline and displays energy options from the user’s utility before they have to enter any information. The team’s hypothesis was that by displaying energy options and cutting out the step of having to enter the ZIP code first, users would essentially be able to instantly “window-shop” by seeing what’s available in their area immediately upon landing.

 Creative Sample #4: Treatment (after) paid traffic landing page for energy shopping website

The A/B split test ran for about four weeks and the treatment improved conversion rates by 32% over the previous version and was rolled out to all paid traffic coming to ElectricityRates.com.

“Give your customer what they're looking for as soon as they land on your site. Eliminate any unnecessary steps to make the buying process as smooth as possible and constantly question ‘do we really need this step/process in order to better serve the customer?’” advised Adam Cain, Director of Customer Experience, Power Target LLC. “Also make your landing experience as relevant as possible to the user’s intent including their location, demographics, etc.”

Quick Case Study #3: Six strategies that helped credit union secure 77 backlinks

Over a seven-month span of link building, the team at Truliant Federal Credit Union secured a total of 77 backlinks for the financial institution’s website by testing six different strategies. Of the 77 links acquired, over 70% were to target pages. Here are the strategies they experimented with:

• Unlinked mentions – “Once you’ve identified these unlinked mentions, you can reach out to the appropriate contact and request to update the mention with a clickable hyperlink back to your site,” said Jeff Hibbard, Vice President of Digital Innovation, Truliant FCU. The team identified 1,500+ opportunities for this strategy, securing 35 links with a 2.5% conversion rate.
   
• Broken links or 404 pages – The team identified 250+ opportunities for this strategy, securing 18 links with a 2.6% conversion rate.
   
• Link additions – The team identified pages with existing backlinks where it made sense to offer a Truliant page in tandem with the existing link. The team identified 350+ opportunities for link addition, securing six links with a 1% conversion rate.
   
• Ego baiting – The team found existing external links on the site and tried to secure placement on a Press, About Us, or Featured On page on a third-party website. They identified 70+ opportunities for this strategy, securing two links with a 1% conversion rate.
   
• Thought Leadership – They tried providing quotes, participating in interviews and podcasts, or becoming a regular contributor to relevant publications. The team found 250+ opportunities for this strategy and secured 14 links with a 2.5% conversion rate.
   
• Pitching as a Resource – “This approach requires you to create high-quality and competitive content, you just need to make sure that there is a sincere value add for third-party websites providing a link back to the resource,” Hibbard said. They secured three links with a 0.5% conversion rate.

“In addition to ranking keyword and traffic growth, we also saw improvements on the Page Authority (PA) score from Moz on the same pages,” Hibbard said.

Related resources

A Behind-the-Scenes Look at Creating an A/B test

Marketing Experiment: Learn from our split testing mistakes

Popular conversion optimization, A/B testing and value proposition content

Designing Hypotheses that Win: A four-step framework for gaining customer wisdom and generating marketing results

Subscription Marketing: 4 mini case studies of recurring revenue products