Understanding Customer Experience: 3 quick marketing case studies (including test results)

This article was published in the MarketingSherpa email newsletter.

“Separate the means from the end. Focus on the customer’s experience of the value,” Flint McGlaughlin taught in Value Proposition Strategy: Everything you need to know in 6 minutes.

To give you ideas for focusing on the customer’s experience of the value, in this article we bring you three quick case studies.

First, an online tutoring provider uses two types of online testing to validate its understanding of its customers’ experience. Then, the Canadian Red Cross tests an expression of personal appreciation to its “customers” (i.e., donors). And finally, a recruitment firm formulates its customers’ experience in a defined value proposition for the very first time, and conducts offline tests in customer pitches to determine its effectiveness.

Quick Case Study #1: How online tutoring provider increased conversion rate by 28% with call-to-action (CTA) testing.

The GoStudent performance marketing and conversion rate optimization (CRO) team wanted to better understand how consumers are engaging with their website content in the United States. “We want to make sure the landing pages we design capture the user’s attention, improve brand consideration, and ultimately drive action,” said Colton Weaver, Performance Marketing Manager, GoStudent.

While looking at their site conversion rate and engagement rate for their primary landing page, they noticed many American consumers were visiting the landing page but either bouncing or not converting at as high a rate when compared to their counterparts in other nations.

Creative Sample #1: Primary landing page for online tutoring provider

The team hypothesized that adding a stronger CTA message with action words would drive a higher conversion rate when compared to softer messages currently being used. “We leaned into bandit testing to optimize our CTA while highlighting our free trial messaging to capture the customer’s attention which may have been missed historically,” Weaver said.

(Technical Note: I don’t want to get too deep into the complexities of the statistics behind marketing experimentation, but for the purposes of this article it is important to understand that bandit testing, also called multi-armed bandit testing, is an attempt to minimize the traffic that goes to low-performing variations in an experiment by shifting traffic to better-performing variations. A/B testing usually splits the traffic evenly throughout the test to the different treatments.)

To test this hypothesis, the CRO team initially used a four-arm bandit test to test four variations of the CTA – the control and three variations. These CTAs were:

• REQUEST A CALLBACK [Control]
• BOOK A FREE TRIAL SESSION [Variation #1]
• SIGN UP FOR A FREE TRIAL [Variation #2]
• GET YOUR FREE TRIAL [Variation #3]

Creative Sample #2: Variation #3 of button copy for online tutoring provider

This test ran for two weeks before reaching statistical significance. Variation #3, “GET YOUR FREE TRIAL,” had the strongest performance driving a 13% increase in sign-up rate compared to the control. Variation #2 also showed an improvement in conversion rate (CVR), but not at the same rate as Variation #3. The results for this four-arm experiment are shown below.

Creative Sample #3: Call-to-action experiment results

After analyzing this data, the team decided to conduct a strict A/B test with the control and the winning variation. It is the team’s general practice to rigorously assure statistical significance before implementing a sitewide change. The A/B test ran for two weeks, and Variation #3 produced the following results as compared to the control.

• Change in Conversions: +26%
• Change in Conversion Rate: +28%
• Change in Bounce Rate: -0.3%
• Change in Engaged Sessions: +13%

(Metric Definition Clarity, Engaged Sessions: This user engagement metric shows that your app screen was in the foreground, or your web page was in focus. When your site or app is running but no page or screen is displayed, Google Analytics doesn't collect the metric. The metric can help you understand when users actively use your website or app.)

Looking at the conversion rate of the control over the course of the two tests, during the first test the conversion rate was 7.76%.  During the second phase of the test the conversion rate declined -17% (averaging ~6.6%).  This difference in conversion rate influenced the performance results as Variation #3 remained relatively constant at ~8.5% conversion rate.

Through these two tests, the team found that a more straightforward call to action – “GET YOUR FREE TRIAL” – helps drive site engagement as the consumer spends more time on the site to understand the brand, product, and services as they prepare to sign up for a free trial. However, with shifts in seasonality and macro-economic trends, the team will be testing this CTA again in the coming months to validate this data further.

“Looking at these results, we know that there is a lot more we can be doing and will leverage A/B testing more rigorously in the future months to come. We are excited to have the agility to test and learn easily which enables us to share with pride across all of our global markets. This is just the beginning,” Weaver said.

During this test, the team also learned that making messaging consistent at each stage of the consumer journey helps drive overall brand continuity and conversion rate. The sales team often uses the free trial message to convert leads into customers.

Quick Case Study #2: How Canadian Red Cross earned 11% uplift in donations by leveraging first-party donor data

The customer experience team at the Canadian Red Cross is the steward of long-time donors to the humanitarian charitable organization.

To celebrate World Humanitarian Day during the pandemic, the team wanted to send an email to thank long-time donors and highlight the good work made possible by their contributions. They wanted a high-value touchpoint for loyal donors that give more than average.

“The Canadian Red Cross team created a personalized video email campaign to steward a segment of long-time donors…we tested the personalized video against a control segment and a generic segment who received the same video without personalization,” said Andrew George, Director – Direct & Integrated Marketing, Canadian Red Cross.

 The experiment consisted of:

• Control – Regular direct mail packages
• Treatment #1 – Generic video
• Treatment #2 – Personalized video

“After the campaign, we saw an 11% lift in year-end giving between the personalized video and control group,” George said.

The personalized video generated an 11% uplift in giving over the control group and a 7% uplift in giving over Treatment #1 – even though there was no direct call-to-action asking for donations. The nonprofit earned a 17x ROI for the personalized video compared with the control group and 13x ROI for the personalized video compared to the generic video. Let’s take a look at the creative samples for the winner…

The email’s subject line was personalized – “<first name>, we made this video for you" and the email was signed by CEO Conrad Sauvé.

Creative Sample #4: Canadian Red Cross email introducing personalized video

The email drove people to a simple landing page that hosted the personalized video.

Creative Sample #5: Canadian Red Cross landing page with personalized video

The personalized video mentioned the donor’s name in the audio track at the very beginning, and also had several visuals that included the donor’s name with thank you messages throughout the video.

“The process to execute the videos was seamless,” George said.

Creative Sample #6: Screen grab from Canadian Red Cross personalized video

"To thank long-time donors, the Canadian Red Cross knew it needed to be personal, but they also wanted to show donors how their contributions make a real difference. By sending a dynamic video rather than, say, a letter, they combined the power of visual storytelling with personalization. This is a proven recipe for loyalty, and it worked here, increasing engagement and donations. As an added benefit, videos are easy to share on social media, helping spread the word further,” said Yaron Dishon, CRO, Idomoo (the Canadian Red Cross’s video personalization platform).

Quick Case Study #3: How recruitment firm defined its value proposition, increased pitch wins 35%

I’m always hesitant to write a case study about value propositions, because MECLABS Institute (MarketingSherpa’s parent organization) has a well-known and well-regarded value proposition methodology, so readers sometimes assume that any value prop we publish in a case study is golden.

This next case study isn’t meant to show you a perfect value proposition. The messaging this organization used could likely be improved (as can the messaging in every case study we bring you) because case studies aren’t meant to give you the copy-and-paste answer, they are meant to inspire you to ask the right questions. These aren’t thought leader opinions about what you should do with your brand, they are real stories about what really worked (and sometimes what really didn’t work). How marketers and entrepreneurs just like you, hard-working people bogged down by day-to-day organizational and client concerns, found a way to produce results for their brands and better serve their customers.

This next case study is another such story. I don’t bring it to you because I think the value proposition messaging is perfect, I bring it to you to show the impact on results a company can achieve by moving from not having a defined value proposition to actually defining that value prop. To inspire you to get your organization to define its own value prop. So let’s get into it.

Nigel Wright Group did not have a defined value proposition. Teams sold on their ability to convince clients that the recruitment firm was good at what it does. There was no consistency between teams and each person used a similar but different approach.

The marketing team realized the recruitment firm needed to define its value proposition. They took the approach to do it internally rather than ask their clients. They involved experienced recruitment consultants from across the business in the workshop (representing the best internal client/market knowledge) to explore “Why clients buy from us?”

After capturing many reasons why clients buy from the firm, these were prioritized into the top four reasons by finding the strongest “proof-point differentiators.”

“These were not based on opinions. These were factual points that made Nigel Wright Group unique within the target market that the value proposition related to. It resulted in a powerful proposition that Consultants were then trained to use in pitches that, as a result, became better structured and more effective at winning new clients,” said Justin Barlow, Marketing Director, Nigel Wright Group.

Here is the value proposition the team defined for Nigel Wright Group:

Our clients trust us to quickly attract the highest calibre and most suitable candidates the first time.

Proof points were summarized into four significant differentiators. Each is supported by demonstrable client/market examples and case studies (supported by statistics) that reinforce each differentiator:

1. We access candidates that others can’t:

• We know our market with over 50 UK consultants, 1,000 weekly market conversations and 4,000 candidate meetings a year.
• Our one-team approach means we share information and candidates
• We have the most extensive networks that are other people’s headhunts

2. Our customers believe we enhance their employer brand:

• Our market expertise is unrivalled after 30 years and with 300 years’ regional experience in between our consultants
• We understand our clients and tell their stories accurately to attract the right candidates
• We produce market leading candidate attraction campaigns with world-class marketing

3. We control the process and manage risk, allowing you to focus on your day job:

• We take a thorough consultative brief and then deliver on our promises
• We openly communicate internally and externally so clients’ and candidates’ expectations are met
• We are fully GDPR compliant to ensure that our clients comply, too

4. We deliver sustainable results that produce long-term value

• We provide service excellence assurance – consistently
• We create greater value than our competitors with an open market perception survey ranking us above all competitors across relationships, service, knowledge, quality, and capability.

Once defined, the team sought to test the value proposition by pitching it to clients in real pitches. They also sought feedback from the pitched clients – whether they won or lost – to find out specific elements of the strengths/weaknesses in the proposition.

They discovered that all wins were directly related to the company’s value proposition messaging.

Most losses, however, were price related. Price was not an emphasis in the defined value proposition. The team focused on the value the recruitment firms creates rather than the cost of its fees. The team felt that some clients have a psychological ceiling that they will not accept breaching when it comes to cost.

Overall, they found through this testing that the value prop proof points that support each reason to use the recruitment firm were strong. The team saw a 35% increase in pitch wins using the value proposition messaging in the first six months.

They have decided to revisit and update the value proposition every six months, as they grow and create more value with their business.

Related Resources

Boost conversion. Learn from 150+ CTA experiments.

Customer-First Marketing: Understanding customer pain and responding with action

5 mini case studies about understanding and serving the customer

Marketing Research Chart: Does a good customer experience really affect business success?