By Anne Holland, President
I wanted to write this article personally because I have to admit something that isn't pretty -- especially as the leader of a research firm that prides itself in best practices.
Over the past 68 days we've been conducting an in-house study of what works in email design … and, unfortunately, we didn't use best practices to do it. (More on that below.) As a result, the data is not as trustworthy as it should be. That said, it still demonstrates a significant enough lift that I believe the test has some merit.
So, if your newsletter uses traditional two-column format (one fat column for content and one thin column for clickable navigation and/or advertising) or you run ads in two-column newsletters, keep reading …
Why test left versus right column design?More than three years ago, we were conducting Case Study research on a B-to-B online ad campaign when the marketer said, "You know, the ads on X site got incredibly great click rates, but ads in the same site's newsletter got almost nothing."
The site in question was (and is) one of the top five business news publications in the world, so I was extremely interested in these findings. Why would a banner ad on an already-busy Web page get so much higher click rates than a similar ad in the site's cleaner-designed newsletter? Even accounting for page views vs open rates, the difference didn't seem warranted.
The marketer in question had her own theory: "They put newsletter ads in the right-hand column. That column gets cut off if you're reading email in your Outlook preview, so no one sees the ads. I bet that's part of the problem."
Then last fall EmailLabs announced study results that seemed to give this theory reality. Turns out the majority of at-work email users read either all or part of their messages in the preview pane. And, yes, most preview panes cut off all or most of that far right column.
This worried me tremendously as a businessperson because although MarketingSherpa does not sell advertising (we're a research firm after all), we rely on house ads in our newsletters to help generate sales of our research reports and Summit tickets.
If people weren't seeing our ads, they might not buy our offerings. Were we leaving money on the table due to preview pane "blindness"?
How we conducted our own in-house design test - and what you should do differentlyIf you read MarketingSherpa, you'll know the correct way to test a theory is to conduct either an A/B or multivariate test where you split your list into randomly chosen segments (aka "nth name") and mail different versions of the same newsletters to each slice. Then you watch, with all other factors being equal, what kind of lift or dip you get from the factor you're testing.
We didn't do that.
Typical of most entrepreneurial company leaders, I'm incredibly impatient. When the marketing and newsletter design team asked for 30 days to design the test perfectly and set things up to run it the right way, I said, "Forget the perfect test. Let's just flip the switch now!"
Because I'm President, I won the argument.
On July 24, 2006, the email production team launched our newsletter without prior testing. (See link below to before and after designs.) We made three changes, only the first of which should have significantly impacted ad clicks:
#1. We switched the ad column from the right to the left, which I predicted would help increase ad visibility and therefore clicks.
#2. We removed a small horizontal line of extraneous hotlinks that had appeared across the top of the newsletter … only a tiny handful of people had ever clicked on them, and I thought anything that cluttered design was bad for us.
#3. We changed the headlines from black "ink" to blue (thus indicating clickability) and made them clickable. However, we track headline and story clicks separately from ad clicks so that would not impact ad results tracking.
Over the next two months, we sent seven weekly newsletters on the same schedule we had in the past, to the same house opt-in lists, on the same topics. The ads were for similar products and in similar formats as we'd used in the past, although not always identical to past ads.
Results from MarketingSherpa's design test: the good and the badWe measured the house ad clickthrough rate for the seven weeks before change and the seven weeks after the change. So that's roughly 50 issues before the change and 50 issues after the change.
Ad click rate as a percent was based on measured emails "delivered." This means the emails sent minus the hard bounces (bad addresses or email accounts not currently receiving email due to vacations, etc.)
Result: in the seven weeks prior to the change, our house ad click rate averaged 2.62%. In the seven weeks after the change, our house ad click rate averaged 3.36%.
That's a lift of 28.24%!
Our conversion rates on these clicks held steady or gained slightly (depending on the offering.) Unit sales during the same time period rose by 41%; however, I don't consider that as viable a number to base decisions on because in the "before" period, some of the ads had been for a free offer -- thus resulting in no immediate sales for a few issues. (Again, this is why an A/B test would have been better.)
We also had one unexpected result -- increased reader mail.
Normally, we get *a lot* of reader mail. Hundreds of emails, phone calls and posted comments per day. However, they usually have questions or feedback about our information, not our design. Do readers even care about design? Ours do, more than I expected.
Jennifer Meyer, Marketing Information Manager of Universal Forest Products Inc., was very first reader to write in on July 25th. "I like your new email design. I’d love to see an article/case study on the changes that you implemented and why you implemented them," she said.
Hers was one of the few complimentary letters we received. Dozens more contacted us, but many were not as happy.
"I don't know what it is, but your new email newsletter is less readable," wrote Daken Ariel at Coastway System Technology Ltd. in Vancouver. "I always used to look at your email. Now I hardly do. No change in interest -- but something about the left column links gets in the way of reading it. I assume you A/B tested your new design for open rates or clickthrough rates? I expect you get less readership due to the layout change. Just a comment."
Scott Murray of Esurance Inc. emailed us, "Your recent email redesign added a new column of information to the left of the main message content. This results in your (well-written) headlines being consistently cut in half on my display, forcing me to scroll over to the right or maximize the window to read your headline and summary.
"I was surprised to see this change, given all your recent reporting on 'preview pane' redesigns. This new design has made receiving and reading your newsletter an exercise in patience, as it seems to go against all the guidelines you have suggested of late. Please consider at least moving that new sidebar to the right of the main content. I bet if you test that, you'll see improved clickthroughs as a result -- and happier subscribers."
As you can imagine, while I was thrilled by increased ad clicks and sales, I was despondent over lowering reader satisfaction.
I had a tough decision to make. Should we do the best job we could as a business of pleasing our shareholders who wanted more house ad clicks or should we do the best job we could as an information provider of pleasing our readers who wanted to see the stories without the bother of the ads? Sometimes the twain cannot meet.
We have certainly turned down business deals (including loads of big-money advertisers) in the past because we felt they didn't match our mission of bringing the professional marketers of the world practical data on what really works. If you don't stick to your core mission, you'll ultimately fail as a company.
On the other hand, hey, all we were asking was for readers to open up their issues entirely (or scroll a bit in Preview) to see the story. It's not all that much in the scheme of things.
I figured if that slight reader inconvenience meant we were able to hire more researchers and do an even better job of providing practical information, well, heck, it did meet our mission after all.
My decision became obvious -- stick with the new design.
And, I'm happy to say, we've just added two more editorial and research staff to the team in part due to that decision. So, although readers may be a little inconvenienced, by golly, they'll continue to get even better information from us!
And that's what it's all about.
Useful links related to this articleCreative samples of before-and-after versions of MarketingSherpa newsletters:
http://www.marketingsherpa.com/cs/mstemtest/study.htmlHotlinked list of all the vendors MarketingSherpa relies on for email and Web publishing related services:
http://www.marketingsherpa.com/vendors.htmlMarketingSherpa
http://www.marketingsherpa.com
