by Adam T. Sutton
, Senior Reporter
Personalization is a tried-and-true email tactic. Marketers have improved results by adding first names to subject lines and greetings for years. But is the tactic a little threadbare?
Amanda Gagnon, Education Marketing Associate at AWeber, was not sure if today's subscribers cared that an email addressed them by name. Furthermore, her team at AWeber, an email marketing software provider, speaks to marketers. Hadn't this group been familiar with personalization for years?
"I personally considered this something that our savvy subscribers would roll their eyes at," Gagnon says. "Personalization has sort of been done to death."
Gagnon knew that personalization improved results at other companies, but she and her team were skeptical that it could work with AWeber's audience.
The team tested personalized subject lines in seven consecutive emails in February. Here's how the team set up the test so it could be certain whether personalization boosted response.
Step #1. Select an email list
The team focused this test on its blog subscribers, people who had opted-in for biweekly emails with links to AWeber's most recent posts. The test was part of an ongoing effort the team underwent to improve these emails.
"We had taken basically the same approach [in the blog email] since we started," Gagnon says. "We wanted to see what could optimize our responses."
Set a test schedule
Rather than testing one email, the team chose to test seven consecutive emails sent in February. This would provide a clear indication of whether personalization would improve results.
The blog email is delivered on Tuesdays and Thursdays. The test emails were sent on the following days in February: 2nd, 7th, 14th, 16th, 21st, 23rd, 28th.
Step #2. Select a single variable to test
The team planned to send a basic A/B test on each of the days listed above. A random half of its list would receive a normal, unaltered email (email A), and the second half would receive a test email (email B).
The team chose one part of the email to test: the subject line. The only difference between the emails was that the subject line for email B included the first name of the subscriber as the first word.
For example, here are the two subject lines for the email sent on Feb. 28
- Email A: "Email Marketing Advice From 2 Guys (Who Know What They're Doing)"
- Email B: "[First Name], Email Marketing Advice From 2 Guys (Who Know What They're Doing)"
The team could have personalized the body of the email, or tried variations on how the subscriber's name was referenced in the subject line. But that would have made the results less certain. By focusing on a single change to a single part of the email across multiple sends, the team could draw definitive conclusions from the results.
Step #3. Double check your data
Many people mistype their names on email registration forms (whether on purpose or by accident). If you're testing personalization, you need to ensure the words you pull from your database do not embarrass your brand.
Clean up your data before launch by correcting or removing the following:
- Capitalization errors
- Offensive language
- False names ("Notgiving Myname")
- Obvious misspellings
Be careful not to overcorrect, though. You might find some subscribers prefer exotic spellings over traditional.
Tests have to be valid
A/B testing helps reveal answers, but those answers are only as valid as the tests themselves. There are several threats to the validity of your tests, and one is statistical significance.
Statistical significance depends on sample size (the number of emails you're sending in the test) and the number of positive or negative responses to each treatment. Only tests that achieve a "statistically significant" difference in response between versions A and B are valid.
Check the "useful links" section below for more information about validity, statistical significance, and a free validity tool from our sister company, MarketingExperiments.
* Note: We strive to take an even more rigorous approach when reporting on case studies of companies that have a vested interested to be in front of our audience, such as marketing agencies and vendors. That's why you'll see a laundry list of metrics below, including the open and clickthrough rates for each email tested in the campaign.
Gagnon was somewhat surprised to see that the personalized emails had a 5.13% higher average open rate than the regular emails. But she was really
surprised that they also had 17.36% higher clickthrough rate (results of all seven emails are below).
"Clicks actually blew opens out of the water," she says. "It turned out that was where the personalization seemed to have the biggest effect."
Results from the 7 tests
The open and clickthrough rates were higher for the personalized emails in every test. Feb. 2Subject line A
: PadiAct Takes Your Targeted Subscriptions to a New LevelOpen rate
: 2.41%Subject line B
: [First name], PadiAct Takes Your Targeted Subscriptions to a New LevelOpen rate
: 3.03%Open rate increase
: 6.67%CTR increase
: 25.31%Feb. 7Subject line A
: Improve Deliverability In Two Simple StepsOpen rate
: 3.07%Subject line B
: [First name], Improve Deliverability In Two Simple StepsOpen rate
: 3.39%Open rate increase
: 4.71%CTR increase
: 10.54%Feb. 14 Subject line A
: Don't Kill the Romance: 7 Email Marketing Buzzkills to AvoidOpen rat
: 3.1%Subject line B
: [First name], Don't Kill the Romance: 7 Email Marketing Buzzkills to AvoidOpen rate
: 3.68%Open rate increase
: 7.15%CTR increase
: 18.76%Feb. 16Subject line A
: Could An Email Campaign Be Your Business?Open rate
: 2.57%Subject line B
: [First name], Could An Email Campaign Be Your Business?Open rate
: 2.81%Open rate increase
: 2.11%CTR increase
: 9.43%Feb. 21Subject line A
: Bending the Email Best Practices Rules Open rate
: 2.17%Subject line B
: [First name], Bending the Email Best Practices RulesOpen rate
: 2.6%Open rate increase
: 7.44%CTR increase
: 20.05%Feb. 23Subject line A
: Effective Marketing: It's All About Your SubscribersOpen rate
: 1.86%Subject line B
: [First name], Effective Marketing: It's All About Your SubscribersOpen rate
: 2.33%Open rate increase
: 2.89%CTR increase
: 25.27%Feb. 28Subject line A
: Email Marketing Advice From 2 Guys (Who Know What They're Doing)Open rate
: 2.83%Subject line B
: [First name], Email Marketing Advice From 2 Guys (Who Know What They're Doing)Open rate
: 3.18%Open rate increase
: 4.97%CTR increase
Subject lines impact CTR
If the team only changed the subject lines in each test, then why did the clickthrough rate increase? Aren't subject lines supposed to only affect open rates?
"My theory is mindset," Gagnon says. "People decide whether or not to open an email based on so many factors, and what the subject line says, I think, has a far smaller effect than how much time someone has available."
In other words, a personalized subject line cannot give subscribers more time, but it can influence the people who would likely have opened the email anyway, Gagnon says. Seeing their first names got subscribers more interested in the email and more willing to click.
The team will also use personalized subject lines more often going forward, but not for every message, she says.
"We're actually going to test this more," Gagnon says. "I want to do personalization in the body of the email as well."
Useful links related to this article
Basic Validity Tool DownloadBad Data: The 3 validity threats that make your tests look conclusive (when they are deeply flawed)Email Research: The 5 best email variables to testQuick Lifts: 4 ideas to increase email clickthroughEmail Marketing: Helzberg Diamonds garners 288% sales lift with animated, personalized promo Research Update: The state of email marketing testing and optimizationOptimization Summit 2012 in Denver: Measure, Test, ConvertAWeber
- Feb 28 email
- Feb 21 email