Email Tests: 100% to 400% improvement for website relaunch announcement

by Adam T. Sutton, Senior Reporter

Mike Cieri, Product Manager, Digg, and the social news website's team were finishing off a major website redesign in July. They planned to launch the following month and needed to call major attention to the new site.

The updated Digg included a sleeker design, more personalization and simpler sharing tools. The team felt the revamp would be a big hit with active Digg users and wanted to make sure they were the first to know.

But, Digg also had a large number of in-active users -- people who created a Digg account but had not visited the site for six months or longer. The team wanted to alert these members that Digg had changed and that they should return.

"The fact that we have a major platform launch gave us a nice opportunity to have a touch point with some of our users who'd not been to Digg for a while," Cieri says.

Contacting Digg members with a standard marketing email to urge visits might have helped, but the team needed more than that. It needed its email marketing to pull more than its usual weight to help Digg's launch.

CAMPAIGN

The team planned to test multiple emails in a small percentage of three audience segments. The strongest performing emails would be sent to the remainder of each segment. This approach ensured that the team used the most effective emails to reach its audience and encourage visits to the new site.

Here are the steps the team followed:

Step #1. Segment members by 'recency'

Digg's large audience had a variety of members; some of whom visited the site daily and others who had not visited in 18 months. These members had different relationships with Digg and needed to be addressed differently.

Cieri's team broke its audience into three large segments:
o Active members - those who had visited the site within the last two months
o Lapsed members - those who had not visited for two to 18 months
o Very lapsed members - those who had not visited for 18 months and longer

Digg uses a member account system, making it easy to track the members' visits. Visitors to Digg must have an account to use the site's core functions such as article voting and submission. All visitors can read articles.

Step #2. Design test emails for 'active' group

Cieri realized Digg's most-active members were the most likely to first see the redesigned website, so his team set out to reach them first and do so before they saw the site.

The team designed four emails to test on a sub-segment of this group. The emails differed in their imagery and messaging. For example, one highlighted the site's new features, while another had a more celebratory tone around the site's launch.

Each of the four emails announced Digg's redesign, urged members to visit and also included:
o Image-based format
o Message short enough to be viewed on a single screen (no scrolling)
o Single green button with a call to action, such as: "Start Digging"
o Two sentences of copy

"Our design team really got into it and came up with a range of different options in terms of the imagery and text and subject lines that we could use," Cieri says. According to him, Tyson Rosage, Designer, Digg, played a central role in creating the emails' sleek, graphical designs..

Step #3. Test emails on a small percentage of the 'active' group

The team needed to determine which of the four emails would be the most effective at pulling Digg's active members to the new website.

The team targeted 2 percent of its 'active' members list and divided the test emails evenly among the sub-segment. The site launched on Aug. 25. About one week later, in early September, the team sent the four test emails and monitored their results.

- Test size depends on audience size

Be sure when setting the size of your test audience to select a group large enough to achieve statistically valid results, otherwise you'll be basing decisions on shaky data.

Some testing platforms automatically incorporate this calculation. Marketers who are less-fortunate should do the math to make sure their tests are large enough (see useful links below).

"The numbers were there to the point where we were comfortable working with the results we got and to have a good idea of how the emails would perform with the larger groups," Cieri says.

Step #4. Send top-performing email to remaining 'actives'

About one week after sending the test emails, the team identified the winner (see creative samples) and started sending it to the remainder of the 'active' members list. The email included:
o Frankenstein theme
o Subject line: "We've Created A Monster - The New Digg Is Here"
o Announcement: "Introducing the new Digg."
o Call to Action: "Start Digging"

Mailings were scheduled over a period of about three weeks. The team could have sent every email at once, but, Cieri says they felt it was more prudent to gradually alert the audience.

This allowed the team to remedy any glitches in the new site before exposing its entire active audience to them. It also avoided a massive, one-time influx of visitors and instead promoted a steady stream of increased traffic for several weeks.

Step #5. Repeat the process for the two remaining segments

The team created four unique test emails for each of its two remaining lists, the 'lapsed' and 'very lapsed' members. Starting with the 'lapsed' members, they followed the same process: test, select a winner and gradually mail each list.

The team started this process with its 'active' list since its members would be the most likely to visit the site in the near term. Also, any glitches in the new site could be resolved before the team reached out to inactive members and welcomed them to the new Digg.

The winning 'lapsed' member email (see creative samples) included:
o Zombie theme
o Subject line: "Resurrect yourself on the new Digg"
o Announcement: "A lot has changed since you were last on Digg."
o Call to action: "Come Alive"

The winning 'very lapsed' member email (see creative samples) included:
o Couch-potato theme
o Subject line: "Look Alive. The New Digg is here."
o Announcement: "Follow your sources. Get your news on the new Digg."
o Call to action: "Start digging"

The entire process took about two months and ended in late October.
RESULTS

"So far the results have been pretty good...We have seen a good amount of lift and a lot of people coming to check out the site. So far, we’re happy with it," Cieri says.

"We're also happy with how many of our 'lapsed' and 'very lapsed' users have responded -- the open and click rate for our most inactive members more than doubled what we see on our marketing emails."

'Active' member email:
o Open rate: 42% (5x increase over typical Digg marketing email)
o Clickthrough rate: 13% (4x increase)

'Lapsed' member email:
o Open rate: 20% (2.5x increase)
o Clickthrough rate: 7% (2x increase)

'Very lapsed' member email:
o Open rate: 15% (2x increase)
o Clickthrough rate: 6% (2x increase)

Useful links related to this article

CREATIVE SAMPLES
1. 'Active' members email
2. 'Lapsed' members email
3. 'Very Lapsed' members email

Members Library: Sherpa 101: How to use Digg to go viral -- 10 Strategies

Sample Size Calculations for Marketing Testing

Digg