Close
Join 237,000 weekly readers and receive practical marketing advice for FREE.
MarketingSherpa's Case Studies, New Research Data, How-tos, Interviews and Articles

Enter your email below to join thousands of marketers and get FREE weekly newsletters with practical Case Studies, research and training, as well as MarketingSherpa updates and promotions.

 

Please refer to our Privacy Policy and About Us page for contact details.

No thanks, take me to MarketingSherpa

First Name:
Last Name:
Email:
Text HTML
May 22, 2008
How To

Test Results: Will Reminder Date in Subject Line Increase Article Readership?

SUMMARY: If you’re a subscription site that gives non-members limited access to certain content, can you encourage readers to take advantage before the allotted time expires?

MarketingSherpa recently tested adding an expiration date to our own newsletter subject lines to see if it might encourage more opens and clicks, before our articles went behind the members-only barrier. Read on to find out what we learned.
When you operate a publishing site like MarketingSherpa, you want to be clear to your subscribers about the content available to them. Articles published in our nine weekly newsletters are “open access” to our hundreds of thousands of readers for seven days before they go into a members-only archive. The barrier date is stated in each email.

Still, results from a survey conducted earlier this year showed that 68% of our readers don't know how long articles are available before going behind the membership wall.

We decided our limited-access policy needed to be clarified. We also wanted to see if adding an expiration date to newsletter subject lines might create a sense of urgency that would boost open and clickthrough rates.

Seems like a no-brainer, right? Never in a million years did we think it would do the opposite. Well, it did. And the lessons we learned are something all marketers can appreciate.

Here are the four steps we followed to conduct the A/B split test and measure results:

-> Step #1. Develop new subject line style

Our standard subject line style starts with the name of the newsletter -- B-to-B, B-to-C, ContentBiz, EmailSherpa and so on -- and includes a few words about the article’s content.

For the test, we added an expiration date in brackets at the end of the subject line for the test groups. This way readers scanning their inboxes could immediately see how long an article was “open access.”

Example:
o Control - B-to-C: Personas Primer
o Test - B-to-C: Personas Primer [Expires 3/26/08]”

-> Step #2. Randomize delivery

Each week, our email production team sent roughly half our newsletter subscribers the articles with the test subject line; the other half got the control. We randomized the delivery options so the tests went to different groups of subscribers each week.

-> Step #3. Provide test ample time

We decided the A/B test needed enough time to generate results that weren’t skewed by the reactions of subscribers to a different subject line style. So, we ran the test for 10 weeks.

-> Step #4. Track key metrics

To determine whether adding an expiration date boosted reader engagement with articles, we tracked five metrics:
o Open rates
o Clickthroughs to the article link
o Total newsletter clicks, including house ads, our jobs posting and all other secondary links
o Email bounce rate
o Unsubscribes

In addition to tracking total figures, we broke out these metrics for the nine different newsletter titles to track potential differences in behavior based on each newsletter’s particular subscriber base.

Well, what did we find out? Turns out adding an expiration date to the subject line of our email newsletters didn’t boost opens or clickthroughs. In fact, it had the opposite effect, contrary to our expectations.

Newsletters with an expiration date in the subject line performed slightly worse than those that used our control subject line in key engagement metrics:
o 0.7% decline in average open rate
o 2.5% decline in average article clicks
o 1.1% decline in average total clicks

At least the difference in the subject lines produced no significant difference in bounce rates or unsubscribes.

Looking at the results by newsletter title, the control subject line beat the test line in almost every metric.

Although we had some anecdotal evidence (a few emails from happy readers) supporting the test subject line style, the message in the metrics was clear: Adding an expiration date wasn’t an improvement over our standard subject line.

In the end, the test was another reminder that marketers can’t make decisions based on gut instincts because you never know how your customers will react. Go with the data.

Useful links related to this article

ExactTarget - MarketingSherpa’s email service provider:
http://email.exacttarget.com/


DeliverySuccess - helps with newsletter production:
http://www.deliverysuccess.com/


Comments about this How To

May 22, 2008 - Jakob Nielsen of Nielsen Norman Group says:
As you say, one never knows without testing, but to hazard a guess: Maybe the lower clickthrough rate was caused by recipients getting a false sense of NON-urgency by seeing a deadline 7 days into the future. People might have thought, "Oh, I have plenty of time left, so I will get to this email later." Of course, when "later" came around, your email had dropped out of sight long ago, below the visible number of messages in that user's inbox. Email is such an immediate medium that anything that's not today (or maybe tomorrow) may not invoke much of a sense of urgency.



Post a Comment

Note: Comments are lightly moderated. We post all comments without editing as long as they
(a) relate to the topic at hand,
(b) do not contain offensive content, and
(c) are not overt sales pitches for your company's own products/services.










To help us prevent spam, please type the numbers
(including dashes) you see in the image below.*

Invalid entry - please re-enter




*Please Note: Your comment will not appear immediately --
article comments are approved by a moderator.