CHALLENGE
When Heidi Hess joined Sierra Club as Online Program Manager in 2004, one of her first tasks was to help launch a fortnightly email newsletter.

During that first year, Hess and the email team focused on building the permission list to hundreds of thousands of opt-ins. Plus, they carefully A/B tested subject lines for months to determine what sorts of topics and wording would get the highest open rates. (Note: See link below to a list of the Club's subject lines over time.)

For 2005, Hess' team raised the bar again, this time focusing on what specific sorts of content would get the most clicks. As a result, average click rates leapt by a remarkable 63% over the year.

In fact, the newsletter became such an internally-recognized success that the Web team's content building decisions began to center on would email recipients want to click to this? (The opposite is true of many organizations, where email content is driven by the site, instead of the site being driven by email metrics.)

Just as the email program was riding this peak of success, Hess became aware of study data showing that 69% of Outlook email users view messages in their preview panes and that Hotmail and Yahoo users would soon follow suit.

She quickly tested the preview pane in Outlook for herself. Turned out the Sierra Club newsletter, which was beautifully designed for a full open, looked pretty bad in preview. (Link to screenshot below.)

What to do? Should you mess with proven success?

CAMPAIGN
Hess decided to A/B test (using randomly segmented names) a preview-pane friendly design vs her standard template for several issues over December 2005-January 2006.

The new design looked almost exactly like the old design, except for two big differences:

#1. Removing admin messaging from the very top

In line with the email newsletter best practices of the time, the old design had several lines of admin messaging regarding whitelisting and how to click to view an HTML version online if the in-box version was garbled.

This was presented in tiny type, but still took up a solid swathe of white space at the top of the newsletter. When viewed in an in-box, that seemed like too high of a percentage of real estate for admin.

So Hess asked the design team to move that information further down.

#2. Tacking a "boring" Table of Contents at the top

Hess asked the design team to add a table of contents section directly above the top of the old newsletter template.

Instead of Sierra Club's cheerful orange template and logo, viewers looking at the issue in the bottom preview pane would only see a list of clickable headlines in 12-point Verdana. The background color was white. The headlines were hotlink blue. (Link to sample below.)

When they clicked, viewers would be brought down to the section of the regular, colorful newsletter the headline referred to. From there, they could read a summary and decide whether to click to the Web site to see more.

Hess didn't know what to expect from the A/B tests.

However, as the Web design world Web has discovered, when you change your look dramatically, your regular visitors are often disconcerted. Even if the new look and navigation is far more intuitively-usable, often Web redesigns see a slump in visitor conversions or page views for a month or two until the audience gets used to the new version.

(This is one of the reasons why most Web developers prefer to tweak their way to a new design gradually rather than launching full throttle.)

Would email results undergo a similar plummet?



RESULTS
"The test results weren't terribly dramatic," says Hess. Average open rates didn't budge, but then this was expected because a design change should affect clicks far more than opens. However, click rates held steady, too.

So, Hess took a deep breath and decided to roll out the new design to the entire list for 2006. She figured that even if the redesign didn't help as much as hoped, at least it didn’t hurt.

But then something lovely happened. Month after month in 2006, the click rates began to swing upwards. A newsletter that had already risen 63% in click rates the year before now rose an additional 55%!

We don't know of any popular newsletter that has seen this type of steady dramatic increase for 24 months. Some have held steady, and many have done slightly better … but this leap two years in a row is close to unprecedented.

Hess says she can’t lay all credit to the design, much has to do with content as well -- and that's as it should be. However, she is feeling a bit triumphant these days about her preview pane leap of faith.

What's next? She's considering redesigns to improve the look of the newsletter in specific email clients such as Yahoo!'s revamped email in-boxes. She'll segment the list so Yahoo!-users only will get specifically-focused design for their unique in-boxes. It's a great idea.

Useful links related to this article:

Creative samples of Sierra Club's newsletters and preview pane views:
http://www.marketingsherpa.com/cs/sierra/study.html

EmailLabs -- Sierra Club's email service provider
http://www.emaillabs.com

Sierra Club
http://www.sierraclub.org