April 23, 2024
Case Study

Conversion Optimization Case Studies: A/B testing in the product and on lead capture ads


You don’t need to have all the answers.

You just need to know how to learn the answers…from your customers.

One way to do that is by testing. To spur your next great test idea, in this article we show you how a classifieds website enhanced user engagement and how a lending platform optimized email campaigns.

by Daniel Burstein, Senior Director, Content & Marketing, MarketingSherpa and MECLABS Institute

Action Box

Looking for test ideas? Try this prompt using the Conversion Pro expert assistant in MeclabsAI. It's totally FREE to use (for now):

"Generate A/B testing ideas for [specify the webpage, feature, or product] focusing on [specify the element, such as headlines, call to action, visuals, user flow, etc.]."

And get more test ideas from these case studies…

Quick Case Study #1: Classifieds website adds more daily promotion options, increases revenue 13%

999.md is a classifieds website with, on average, 300,000 people visiting every day to buy or sell products and services.

As a two-sided marketplace, there are both sellers and buyers. Sellers list their items and goods, while buyers search for and buy them.

The site’s revenue is derived from promotional services used by sellers, like the Autorepublisher service. It re-lists advertisements at specified times and days. This service effectively boosts listings to the top during predefined times, contributing significantly to the platform’s overall revenue when compared to other services it offers.

“It is crucial to continually test and seek improvements for this service as it has great potential,” said Aksinia Chumachenko, Product Analysts Team Lead, Simpals (999.md’s holding company).

Let’s take a look at one of the team’s tests to give you ideas for your own product testing.

BEFORE: Limited default hours

The team could only adjust a few parameters for the Autorepublisher, such as pricing and frequency.

Initially, the Autorepublisher was set to activate three times per day. Users had the option to adjust these hours by either reducing or adding more at their discretion.

Creative Sample #1: Control Autorepublisher settings with 3 daily options

Creative Sample #1: Control Autorepublisher settings with 3 daily options

AFTER: Increased default hours

The team analyzed customer usage and noticed that some users were adding extra hours themselves. So in the treatment variant, the team altered the number of hour options per day, adding two additional slots – from three launches a day, they moved to five.

Creative Sample #2: Treatment Autorepublisher settings with 5 daily options

Creative Sample #2: Treatment Autorepublisher settings with 5 daily options

The test ran for two weeks.

RESULTS: Increase in revenue and weekly republications

With the addition of two extra options per day, many users decided not to remove these new slots. This adjustment led to a rise in weekly republications. Because of this modified engagement, revenue also increased since payment is collected for every republication.

Weekly republications went from 12.6% for the control to 16.2% for the treatment – an increase of 22%. Revenue from this service also increased, by 13%.

The support team regularly receives user feedback, and they haven’t seen any negative reactions to the change.

I like this test because it is a great example of a simple change based on user activity that produced a significant lift. I also like it because the team actually tested it and didn’t add more product options arbitrarily. After all, a company could easily be tempted to let the additions get out of hand. If five increases revenue, wouldn’t six be even better? Or for that matter, 24?

When I asked Chumachenko about this subtle threat of overdoing it, she told me, “Indeed, adding too many hours could eventually stop revenue growth or even decrease it if users were to resist the change and not engage with the service at all. That’s why we closely watched the conversion rate. We don’t have a precise hour count where this effect might occur, but it’s about balancing conversions and the resulting revenue.”

“Before conducting an A/B test, it is essential to agree on clear objectives for how and to what extent you want to influence metrics, calculate unit economics, and anticipate potential trade-offs,” she advised.

Quick Case Study #2: Lending platform grows opt-in list with ad testing, increases email clickthrough rate from 2% to 4.3% with flyer-style email

There are two types of MarketingSherpa case studies. There are the ones that you probably assumed would work and aren’t shocked about the outcome. We share those with you to trigger a similar idea in your own head and explore how the team actually executed it. Quick Case Study #1 in this article is a great example.

Then there are the ones that are counterintuitive. If you see the before and after, you would pick the wrong one as the winner. We publish these to challenge your assumptions.

And this next case study is a great example. At least it challenged my assumptions, because I would think the educational-style email would outperform the flyer style. But I asked some pretty in-depth and direct questions, and the story vetted out (frankly, for each case study we actually do publish, there were many other stories that didn’t hold up to scrutiny).

Even better, this story goes a level deeper. When I saw this team had sent 460,100 emails over the past 15 days, I surmised they’ve probably built a pretty sizable opt-in list. No easy task. So I also asked some pretty detailed questions, and they were gracious enough to provide a behind-the-scenes look.

Without further ado, let’s dive in.

BEFORE – Newsletter-style email

Niche Capital describes itself as a ‘consumer-first lending platform’ and promises ‘Personalized Business Capital without the Spam.’

The team was sending a newsletter-style email with the intent of educating their opt-in list on lending products.

Creative Sample #3: Newsletter-style email

Creative Sample #3: Newsletter-style email

But the team wasn’t happy with the open or clickthrough rates, or the delivery rate (below 50%).

AFTER – Flyer-style email

They changed their email format to more of a flyer style and stopped sending the educational-style emails. “We found our opt-in list was much more receptive to this,” said Dayten Rynsburger, Chief Revenue Officer, Niche Capital CO.

Creative Sample #4: Flyer-style email

Creative Sample #4: Flyer-style email

RESULTS – Higher clickthrough and open rates

“Everything is so expensive and with inflation through the roof, market pressure, etc., we found business owners have a lot less time for educational content and really see value in ‘to the point’ style of email. This will not always be the case but in today’s market the simple ‘do you need money?’ is really out-performing the ‘let me educate you,’” he said.

The team realized the following results:

  • Open rates went from ~50% to 87.7%
  • Clickthrough rate went up from 2% to 4.3%

The team also made deliverability changes. "If no one sees the email, no one can click the email," he said.

They noticed a 40% increase in delivery rate when they added rDNS records and did not bulk send on just one IP.  “We rotated already warm IPs and changed to a different MTA called Power MTA [from] SparkPost now MessageBird [which has been rebranded to just Bird],” Rynsburger said. With PMTA the team set ISP rules to ensure they were not over-sending to just one ESP.

“As far as DMARC, SPF, and DKIM, we set all that up with our domains and subdomains via Cloudflare and in our PMTA server,” Rynsburger said.

Good stuff. But I was still itching to go deeper and wanted to know how they built that opt-in list to begin with. Let’s go back to the beginning of this funnel and see how they built it.

A/B testing for permission-based emailing

The team creates their own content. “We put a lot of time into SEO and posting articles frequently to ensure we are number one on Google when people go looking for us,” Rynsburger said. And there are a few opt-in funnels on the company’s website, like on the main blog page. “However, the majority of our email subscribers are acquired using lead magnets and Facebook ads,” Rynsburger said.

The funnel is simple – from ad to landing page to lead magnet deliverable. Core to its success is tapping into a real customer need and delivering value. “Our strategy is very simple: give away something valuable in exchange for an email address,” he said. “It is important the deliverable is good. A bad deliverable is worse than no deliverable.”

Here’s is an example of one of the lead magnet funnels they used to acquire B2B consultants and service professionals.

First, they identified messaging that would resonate with business consultants – effectively using LinkedIn by improving your profile and sharing helpful content on it. “As we all know, Linkedin is a goldmine to find working business professionals and decision makers, however most people us it incorrectly. So, we leveraged this knowledge to add to our list using a Linkedin-centered ad.”

Before we look at the ads, let’s talk about the channel. I asked him why he used Facebook ads for a LinkedIn-focused lead magnet.

“We advertise everything on Facebook. It’s what we know and much more cost-efficient and tried-and-true than LinkedIn. In addition, most people, including business owners, use Facebook or Instagram. People scroll these platforms aimlessly every day, so we come to them where they hang out. On LinkedIn, people log on, do or find what they need, and then log off,” Rynsburger explained. “For the most part, Meta ads are what we know, and if it ain't broke, don't fix it!”

They test all of their ads, one variable at a time. Here’s an example split test where the only variable that changed was the headline.

BEST – Top-performing headline in Facebook ad

In this test, “GRAB THIS FREE LINKEDIN MASTERCLASS” was the top-performing headline.

Creative Sample #5: Best-performing Facebook ad used to grow opt-in email list

Creative Sample #5: Best-performing Facebook ad used to grow opt-in email list

“YOUR FREE GUIDE TO LINKEDIN MASTERY” was in second place, not too far off the winner.

WORST – Lowest-performing headline in Facebook ad

“MONETIZING LINKEDIN THE RIGHT WAY” was the worst, with cost-per-click roughly 4x more than the top performer.

Creative Sample #6: Lowest-performing Facebook ad used to grow opt-in email list

Creative Sample #6: Lowest-performing Facebook ad used to grow opt-in email list

“If I had to guess it is because it doesn’t have the words ‘Free’ or ‘Your’ which are two things people love: free and themselves. The other ads had much smaller variance but ultimately ‘Grab’ was the winner,” he said.

They aim for $0.50-$1.00 CPC max before worrying about the op-tin rate, then shoot for 50-70% opt-in rate once clicks are cost efficient. The team keeps targeting broad and lets the algorithm find their audience.

List-building landing page

“Once a user clicks on your ad, it is crucial that your opt-in page relays the same message your Meta Ad does,” Rynsburger said.

The landing pages page they used are quite simple. Below is the entire landing page. This landing page (again, the winner after testing) achieved an opt-in rate of 65% to 70%.

Creative Sample #7: Email opt-in landing page for previously mentioned ad campaign

Creative Sample #7: Email opt-in landing page for previously mentioned ad campaign

The team only asks for the email address. In their experience, every additional question can lead to a 50% increase in cost per lead.

As mentioned above, they optimize the landing page to get a 50-70% opt-in rate once clicks are cost efficient. They’ve found that using generic email newsletter pages or minimally relevant copy significantly lowers opt-in rates, potentially below 20% or even 10%, as it fails to maintain user interest from the ad.

And since they are not just using a single generic landing page, they track which campaign the new subscriber came in on to help with email segmentation down the road. “Use a platform like ConvertKit to capture and tag your subscribers based on which ad they were acquired from,” Rynsburger advised.

Upon entering their email address on the landing page, the lead receives an email with the lead magnet.

Creative Sample #8: Triggered email for subscribers

Creative Sample #8: Triggered email for subscribers

You’ll notice they encourage subscribers to engage in this very first triggered email – “Please reply YES to let me know you got it!”

Their email list’s unsubscribe rate ranges between 1% and 2%. “Back in the day we cared much less about who was signing up and how. It was more about getting the numbers up,” he said. “Now we are a lot more targeted in our approach. Running up quick numbers always ends up hurting you in the end, and that is true for most things – from social media to email lists. Having the wrong audience subscribing also hurts your email list and deliverability in the long run so we put much more effort into getting the right people to the lists.”

Related resources

Conversion Rate Optimization: 4 quick CRO case studies about optimizing mobile landing pages, a blog, homepages, and a booking engine

Conversion Rate Optimization Tools: Marketing case studies with brands that used 5 types of CRO tools

Conversion Rate Optimization: 4 quick CRO case studies to help you increase revenue, mobile conversion, and site searches

Improve Your Marketing

Join our thousands of weekly case study readers.

Enter your email below to receive MarketingSherpa news, updates, and promotions:

Note: Already a subscriber? Want to add a subscription?
Click Here to Manage Subscriptions