Close
Join 237,000 weekly readers and receive practical marketing advice for FREE.
MarketingSherpa's Case Studies, New Research Data, How-tos, Interviews and Articles

Enter your email below to join thousands of marketers and get FREE weekly newsletters with practical Case Studies, research and training, as well as MarketingSherpa updates and promotions.

 

Please refer to our Privacy Policy and About Us page for contact details.

No thanks, take me to MarketingSherpa

First Name:
Last Name:
Email:
Text HTML
Sep 29, 2006
How To

Test Results: Small Email Newsletter Template Design Change Raises Clicks 28%

SUMMARY: Seemingly small email design changes can have a big impact on results. Two months ago, MarketingSherpa launched a test to see if we could improve our own newsletter clickthrough rates with a pretty small template change.

Turns out we did -- clicks increased by 28.24%.

Here's our notes on the experiment, including before and after screenshots, so you can try this (fairly easy) test on your own newsletters, too:
By Anne Holland, President

I wanted to write this article personally because I have to admit something that isn't pretty -- especially as the leader of a research firm that prides itself in best practices.

Over the past 68 days we've been conducting an in-house study of what works in email design … and, unfortunately, we didn't use best practices to do it. (More on that below.) As a result, the data is not as trustworthy as it should be. That said, it still demonstrates a significant enough lift that I believe the test has some merit.

So, if your newsletter uses traditional two-column format (one fat column for content and one thin column for clickable navigation and/or advertising) or you run ads in two-column newsletters, keep reading …

Why test left versus right column design?

More than three years ago, we were conducting Case Study research on a B-to-B online ad campaign when the marketer said, "You know, the ads on X site got incredibly great click rates, but ads in the same site's newsletter got almost nothing."

The site in question was (and is) one of the top five business news publications in the world, so I was extremely interested in these findings. Why would a banner ad on an already-busy Web page get so much higher click rates than a similar ad in the site's cleaner-designed newsletter? Even accounting for page views vs open rates, the difference didn't seem warranted.

The marketer in question had her own theory: "They put newsletter ads in the right-hand column. That column gets cut off if you're reading email in your Outlook preview, so no one sees the ads. I bet that's part of the problem."

Then last fall EmailLabs announced study results that seemed to give this theory reality. Turns out the majority of at-work email users read either all or part of their messages in the preview pane. And, yes, most preview panes cut off all or most of that far right column.

This worried me tremendously as a businessperson because although MarketingSherpa does not sell advertising (we're a research firm after all), we rely on house ads in our newsletters to help generate sales of our research reports and Summit tickets.

If people weren't seeing our ads, they might not buy our offerings. Were we leaving money on the table due to preview pane "blindness"?

How we conducted our own in-house design test - and what you should do differently

If you read MarketingSherpa, you'll know the correct way to test a theory is to conduct either an A/B or multivariate test where you split your list into randomly chosen segments (aka "nth name") and mail different versions of the same newsletters to each slice. Then you watch, with all other factors being equal, what kind of lift or dip you get from the factor you're testing.

We didn't do that.

Typical of most entrepreneurial company leaders, I'm incredibly impatient. When the marketing and newsletter design team asked for 30 days to design the test perfectly and set things up to run it the right way, I said, "Forget the perfect test. Let's just flip the switch now!"

Because I'm President, I won the argument.

On July 24, 2006, the email production team launched our newsletter without prior testing. (See link below to before and after designs.) We made three changes, only the first of which should have significantly impacted ad clicks:

#1. We switched the ad column from the right to the left, which I predicted would help increase ad visibility and therefore clicks.

#2. We removed a small horizontal line of extraneous hotlinks that had appeared across the top of the newsletter … only a tiny handful of people had ever clicked on them, and I thought anything that cluttered design was bad for us.

#3. We changed the headlines from black "ink" to blue (thus indicating clickability) and made them clickable. However, we track headline and story clicks separately from ad clicks so that would not impact ad results tracking.

Over the next two months, we sent seven weekly newsletters on the same schedule we had in the past, to the same house opt-in lists, on the same topics. The ads were for similar products and in similar formats as we'd used in the past, although not always identical to past ads.

Results from MarketingSherpa's design test: the good and the bad

We measured the house ad clickthrough rate for the seven weeks before change and the seven weeks after the change. So that's roughly 50 issues before the change and 50 issues after the change.

Ad click rate as a percent was based on measured emails "delivered." This means the emails sent minus the hard bounces (bad addresses or email accounts not currently receiving email due to vacations, etc.)

Result: in the seven weeks prior to the change, our house ad click rate averaged 2.62%. In the seven weeks after the change, our house ad click rate averaged 3.36%.

That's a lift of 28.24%!

Our conversion rates on these clicks held steady or gained slightly (depending on the offering.) Unit sales during the same time period rose by 41%; however, I don't consider that as viable a number to base decisions on because in the "before" period, some of the ads had been for a free offer -- thus resulting in no immediate sales for a few issues. (Again, this is why an A/B test would have been better.)

We also had one unexpected result -- increased reader mail.

Normally, we get *a lot* of reader mail. Hundreds of emails, phone calls and posted comments per day. However, they usually have questions or feedback about our information, not our design. Do readers even care about design? Ours do, more than I expected.

Jennifer Meyer, Marketing Information Manager of Universal Forest Products Inc., was very first reader to write in on July 25th. "I like your new email design. I’d love to see an article/case study on the changes that you implemented and why you implemented them," she said.

Hers was one of the few complimentary letters we received. Dozens more contacted us, but many were not as happy.

"I don't know what it is, but your new email newsletter is less readable," wrote Daken Ariel at Coastway System Technology Ltd. in Vancouver. "I always used to look at your email. Now I hardly do. No change in interest -- but something about the left column links gets in the way of reading it. I assume you A/B tested your new design for open rates or clickthrough rates? I expect you get less readership due to the layout change. Just a comment."

Scott Murray of Esurance Inc. emailed us, "Your recent email redesign added a new column of information to the left of the main message content. This results in your (well-written) headlines being consistently cut in half on my display, forcing me to scroll over to the right or maximize the window to read your headline and summary.

"I was surprised to see this change, given all your recent reporting on 'preview pane' redesigns. This new design has made receiving and reading your newsletter an exercise in patience, as it seems to go against all the guidelines you have suggested of late. Please consider at least moving that new sidebar to the right of the main content. I bet if you test that, you'll see improved clickthroughs as a result -- and happier subscribers."

As you can imagine, while I was thrilled by increased ad clicks and sales, I was despondent over lowering reader satisfaction.

I had a tough decision to make. Should we do the best job we could as a business of pleasing our shareholders who wanted more house ad clicks or should we do the best job we could as an information provider of pleasing our readers who wanted to see the stories without the bother of the ads? Sometimes the twain cannot meet.

We have certainly turned down business deals (including loads of big-money advertisers) in the past because we felt they didn't match our mission of bringing the professional marketers of the world practical data on what really works. If you don't stick to your core mission, you'll ultimately fail as a company.

On the other hand, hey, all we were asking was for readers to open up their issues entirely (or scroll a bit in Preview) to see the story. It's not all that much in the scheme of things.

I figured if that slight reader inconvenience meant we were able to hire more researchers and do an even better job of providing practical information, well, heck, it did meet our mission after all.

My decision became obvious -- stick with the new design.

And, I'm happy to say, we've just added two more editorial and research staff to the team in part due to that decision. So, although readers may be a little inconvenienced, by golly, they'll continue to get even better information from us!

And that's what it's all about.

Useful links related to this article

Creative samples of before-and-after versions of MarketingSherpa newsletters:
http://www.marketingsherpa.com/cs/mstemtest/study.html

Hotlinked list of all the vendors MarketingSherpa relies on for email and Web publishing related services:

http://www.marketingsherpa.com/vendors.html

MarketingSherpa
http://www.marketingsherpa.com


See Also:

Comments about this How To

Sep 29, 2006 - Kristen Shue of ClickShift, Inc. says:
Good article - but you could've moved up the link to the before/after sample newsletters and made it a lot more prominent. As it stands, I had to scan through the entire article twice before finally finding the link at the bottom. As you know, on the web people mostly scan the content - they don't read. So it'd be an improvement in usability if you made it easier to find the link and show the A/B test visually, instead of describing it in words.


Sep 29, 2006 - Hiram of CAM Environmental says:
I think you sold out -- but I also don't see how you had much of a choice being a commercial enterprize. The reader will always lose out to increased profits. The best companies find a way to marry the two instead of seeing it as "either/or."


Sep 29, 2006 - Bryan Pinn of SitePosition Search Marketing says:
I think I'd want to do some A/B/C/D testing... with horizontal banner variants in the mix. Remove the page-right cut-off as a problem for reader-friendliness and ad visibility.


Sep 30, 2006 - Rick of Go-Mango Fitness Equipment says:
It would be interesting to see how ads placed horizontally in the middle of the article would do against this new left pane scheme - would solve the problem of readability in the preview pane, right?


Oct 02, 2006 - Bill Scully of Siemens Water Technologies says:
I'd be interested in seeing the long term effects. Is your loyal base going to continue to subscribe and/or open the email over time? If the root problem before was usability (preview pane users not knowing there was an ad), I suggest you look at other ways to include the ads in the body ... sounds like a job for your new research staff.


Oct 02, 2006 - Peter Johnston of Marketing Future says:
I'll lay money that the reason for higher click through was that those who couldn't see the navigation on the right now could and wanted to open it to read an article below the fold. Another problem raises its head, however - many people print out their emails to read them and often a left hand navigation email will cut off the texton the right. The answer - include a small what's in this issue click through section in the top left corner, and a feature article alongside, ideally with a right justified picture (so all of the text prints). Then swop the navigation and ads to the right for the rest of the newsletter. Try it - it works!


Oct 02, 2006 - Alison Chandler of AAAS says:
If I were considering the results of the redesign, I would ask: how many people can't see the whole email at once, due to the preview pane? I use a preview pane, but have a large screen with a high resolution and can see the entire issue at once. I wouldn't jump to make changes again until I had an idea of how many people were forced to scroll. The squeaky wheels are not always part of the majority.


Oct 02, 2006 - SimonB of First Conferences says:
An interesting article - can you tell us what the stats were for click-throughs and read rates for the articles after the design change?


Oct 02, 2006 - Anne Holland of aholland@marketingsherpa.com says:
Our traffic stats from article clickthroughs remained rock-steady throughout the period, except for a freakish blip for one particular story which, invariably, is our second most read story of the year. (Our blog awards final winners announcement.)


Oct 02, 2006 - John Bruce of MAGIC/Advanstar Communications says:
I preview all newletters in my email "Preview Pane". I doesn't matter if a newsletter has advertising on the right side, left side or the top. I can adjust the width and height of my preview pane to accommodate anyone's newsletter. And if the newsletter has information related to what I'm interested in, I'll open it fully anyway.


Oct 03, 2006 - Noel Rodrigue of Consultant says:
Here's my 2 cents worth of an idea to 'complete' the change; you may not like some elements of it because it touches on what I'd call 'fixed design' (i.e. things that wouldn't normally change from issue to issue). I would: 1) drop the double underline below the header, far enough to allow the date to be inserted in the header. 2) drop the 'ad' box by the space required to "slide" the main article's title leftwards which will help lift the article's body higher in the viewer. You might also question whether you want to maintain a link to the article at the end of the Intro paragraphs, given that the main title is clickable and brings the reader to the same place. If you don't maintain said link you will want to move the date of open access nearer the main title.


Oct 04, 2006 - Michael Raia of American Dental Association says:
I personally never use the Reading Pane, partly because of security risks and partly because I'm just not in the habit. Have there been any studies on which style of preview pane people usually use? Obviously Outlook has three choices: Right, Bottom, or Auto Preview. I am amazed that anyone would use the Right preview, which cuts off the majority of any e-mail by default.


Oct 04, 2006 - Anne Holland of MarketingSherpa says:
Yes, as I noted in the article above, one study has been conducted on preview pane usership in the past, indicating roughly 60% of business users use preview panes sometimes or always. We're conducting new studies on this this fall and will bring you results as we have them.



Post a Comment

Note: Comments are lightly moderated. We post all comments without editing as long as they
(a) relate to the topic at hand,
(b) do not contain offensive content, and
(c) are not overt sales pitches for your company's own products/services.










To help us prevent spam, please type the numbers
(including dashes) you see in the image below.*

Invalid entry - please re-enter




*Please Note: Your comment will not appear immediately --
article comments are approved by a moderator.

Improve your marketing

Join our thousands of weekly Case Study readers. Enter your email address below to receive MarketingSherpa news, updates, and promotions:
Note: Already a subscriber? Want to add a subscription?
Click Here to Manage Subscriptions