Close
Join 237,000 weekly readers and receive practical marketing advice for FREE.
MarketingSherpa's Case Studies, New Research Data, How-tos, Interviews and Articles

Enter your email below to join thousands of marketers and get FREE weekly newsletters with practical Case Studies, research and training, as well as MarketingSherpa updates and promotions.

 

Please refer to our Privacy Policy and About Us page for contact details.

No thanks, take me to MarketingSherpa

First Name:
Last Name:
Email:
Text HTML
Aug 14, 2006
Blog Post

The Annoying Imprecision of Email Open Rate Metrics

SUMMARY: No summary available.
By Anne Holland

I've started calling email open rates "measured opens" because just saying "open" alone is so misleading.

The annoying imprecision of email opens as a metric has been a problem for a decade now. Yet, every 18 months or so, a whole new wave of marketers discover it. Then, many of them email me dismayed queries. "Is it true? Is email open measurement really so vague?"

Yes, I write back. Precise email opens are an unknowable metric according to today's technology.

First of all, obviously no text-only email has a measured open rate. Measurement depends on HTML, which doesn't exist in text-only.

Second, if your recipient's inbox is blocking HTML (as an increasingly large number do these days), they may open but they won't see your lovely graphics, nor will the open be reported back to you.

Third, and this appears to be the biggest area of current confusion, when an HTML email is viewed in a 'preview pane,' that absolutely counts as an open.

According to an EmailLabs study, 69% of at-work email recipients always or frequently use their preview panes when sorting through their inbox. These are mostly Outlook users, but in the consumer world Hotmail and Yahoo are said to be offering preview panes to their users shortly.

I've encountered many myths about this preview pane open. Most center on the idea that preview pane opens can be measured separately (or eliminated completely) from "regular" opens.

"What if we move the HTML pixel that reports opens to the very end of the creative where the preview pane won't trigger it?" asked one marketer recently. Nah, sorry, won't work. When an email is open in preview, according to your stat reports it is completely open.

Besides, 33% of preview pane users admit they personally consider the preview pane to be an "open." They rarely if ever open all the way, preferring to scroll up and down in that small box. (Have you told your email creative team yet to make your templates look great in the tiny box?)

On the other hand, for preview pane users, the email opens
automatically when they are reviewing their in-box. I know I've "opened" many messages without intending to as I surfed my in-box. (And then worried that the resulting "open" might tell the junk mailer in question that my email account was definitely interested in more of their crud. Blech.)

Open rate confusion perhaps arises because most email metrics reports look so darn precise with decimal figures and all. Also, most marketers are used to being able to get incredible Web analytics and can't believe the same isn't available for email.

Your email open reports can be used to spot major problems in delivery (if not a single AOL user opened your last newsletter, you know you have a problem.) And, they're not bad for fairly general, broad "health of this list/campaign" reports where you're watching trends over time. Plus, if you're running subject line tests with an A/B split, a dramatic difference can be quite useful.

But the real metrics you should be watching are clicks and clicks-to-conversions (whatever your conversion activity may be ... from pageviews to purchasing.)

Which means it's time to tie your Web analytics -- and if possible other data such as in-bound calls, printed coupon redemptions, brand perception, loyalty, etc. -- to your email campaigns.

Unfortunately, this is a nasty tech back-end project for many organizations, involving silo-ed systems and databases. If email really matters to your bottom line, you'll do it. Otherwise you'll continue to slide along guestimating success based on general open and click trends.

If you are one of the former -- or if you have some wonderful new tech to measure email that I'm not aware of -- please post a comment to this blog.

(Yes, we're accepting posted comments on *all* MarketingSherpa stories and Case Studies now at our site so that you can share your insights with the community.)


Sponsor: Get $300.00 off MarketingSherpa B-to-B Summit:
~~~~~~~~~~~~~~~~~~
Network with your peers in Boston, Oct. 23-24,
or San Francisco, Nov. 13-14:

Case studies from:
B & B Electronics
Siemens
BearingPoint
Allegiance
Everon Technology Services

Learn more:
http://www.sherpastore.com/B-to-BDemandGenerationSummit
006.html?1150

Or call 877-895-1717
~~~~~~~~~~~~~~~~~~
See Also:

Comments about this Blog Entry

Aug 14, 2006 - Carolyn Gardner of cardcommunications says:
Excellent article! I couldn't agree more in terms of the imprecision of open rate reporting. Another huge issue that is impacting the inprecision is the reality that many email clients are defaulting to an images off setting. So unless the recipient clicks to view images, again an opened email is not tracked as an open. According to data from an EmailLabs study conducted last October, 52% of at-work users "always use the preview pane" and 17% "frequently" use it. Of these, 45% "rarely" or "never" download the images in preview. Plus, 48% of surveyed at-work preview pane users rarely or never open email messages all the way. They simply view as much of the message as they can in their preview pane, and move on. Of those who do open all the way, more than 15% never turn on the images, so they keep viewing without your pretty graphics.” Another big reality is the increasing use of handheld devices like blackberry's etc. Again, many people are reading their emails via these devices and of course, it's a text only version which cannot be tracked... I totally agree I that open stats should be referred to as "measured open rate". While we're on this metrics rant, I think it's also important to remember that success should not purely be derived from the opens and click-throughs. When we work with clients at cardcommunications (http://www.cardcommunications.com), we always like to find out if there are other measure of success to be considered. For example, we're working on a viral campaign for a sales recruiter right now. The goal is to receive as many good cv's as possible. So at the end of the day, it's not going to be so much about open rates and click-throughs as it will be about the total number of cv's this company recieves - aka what the actual conversion is. Happy emailing! Carolyn Gardner Founder & President cardcommunications inc. http://www.cardcommunications.com


Aug 14, 2006 - Ophir Prusak of Resperate.com says:
Anne, While I agree that some people over emphasize open rates, I disagree that " ... the real metrics you should be watching are clicks and clicks-to-conversions .." In today's over crowded inboxes, you can't afford to ignore open rates. It is possible to get real value and insights from open rates. Here are 6 tips from my personal experience. 1 - Open rates should be used to test/measure subject lines and deliverability. 2 - Open rates mean very little as individual numbers. They should be compared to other open rates. It does not help much to know that an open rate was 17.4% It is helpful to know that when doing an A/B split on a subject line, Group A had a 17.4% open rate and Group B had a 23.8% open rate. Just be sure the numbers are statistically significant. 3 - When comparing open rates, you should compare numbers from the same list. This doesn't have to be the same actual people, but something that is comparable. For example, comparing the open rate for the people who signed up to your daily list in the past 10 days. 4 - Use open rates to spot trends. If you are sending to the same list with the same-ish subject ("Joe's Newsletter for Aug 2006") and the open rate changes it could be seasonality (summer dips) or deliverability (you just got added to a blacklist). 5 - Break down open rates into domains. In order to really see what's happening, you really need to see what's happening at the domain level. 6 - Test click rates against *both* sent email and opened email numbers. I find this is the only real way to differentiate between the effectiveness of the subject line and the email body (content). If you're only looking at clicks / sent, and not clicks / opened, you're not seeing the whole picture. Lets say you have a list of 200,000 people and do an A/B split for the same offer, but with different subject lines and different email body. Group A Sent - 100,000 Opened - 30,000 Clicked - 9,500 Open Rate = 30% Clicks / Sent = 9.5%


Aug 14, 2006 - Chris Neil of Prospectics says:
I hear the same question and confusion, too. I don't know of any magic tricks to count opens. Some deliverability vendors offer delivery rate stats but they are based on samples. The one thing we do to help with expanding the metrics beyond just clicks is track what we call outcomes. We generate a snipet of html code and put it on "outcome" pages on the website that the email is pointing to. We then track those outcomes back in the email software. This way we see open, pass-along, unsubscribe, clicks and outcomes within the one application without having to integrate web analytics too. For marketers who can post content to their site with out having to involve IT resources, this gives them a way to close the loop on the impact of the campaign without having involve other departments to link multiple systems together.


Aug 14, 2006 - Paige Flory of Listrak says:
Thanks for the interesting article. I wanted to comment to share that Listrak has implemented some technology that can report if the message is left open for longer than five seconds. We report this separately as a "read" to help differentiate those false positives in the "open" report resulting from the use of the preview pane. Of course the click report tells the real story about how readers were engaged and interacted with the message, and if a subscriber is tracked clicking but was not captured as an open or read becasue they received the default text version of the measure, the open and read report are automatically updated. I agree that there is definitely always going to be imprecision in the "open" metric, but I wanted to take a moment to point out how Listrak's technology has addressed some of the issues you mentioned in your article. Please feel free to contact me if you have any questions or want to take a closer look.


Aug 14, 2006 - Allan Hoving of ahoving says:
Hey, wow, I thought I was the only person in the world who still used "Blech." It's my absolute favorite IM response.


Aug 15, 2006 - Liam Mulhall of www.brewtopia.com.au says:
We don't send any email in HTML, so automatically we stand out from the crowd. We don't see open rates but knowing can be more frustrating than not! For us, whom every dollar earned depends on how many people buy a case of our custom label beer from our beerbuilder, tracking variables, copy tweaks, headline alternates, whether the moon is tracking venus etc etc - you end up with analysis paralysis! We must remember we deal with human beings, who have acted the same way for over 2000 years - in large numbers people are predictable. As long as the basics are covered off: - you have built a database of souls who trust your and your organisation and don't mind hearing from you from time to time - you appeal to either their interests or their fears (WIIFM) - you have a way of meeting or beating those interests or fears - and you actually offer something that stands out from the rest of the mob you compete with: Then changing and tweaking and testing will have a zero net effect over the long period with a critical mass - you'll p*ss off as many people as you attract. Improve your 4 points above, increase your size of database and hang around longer and your numbers will increase - it's human nature. I know because thats what we have done over 3 years at Brewtopia and I never tested a bloody thing, other than "would my mum buy this stuff I'm selling from me?"


Aug 15, 2006 - Abbi Rouse of Interfinancial says:
It's funny how these issues burn in our souls even when they're no longer relevant (in my last job my boss refused to measure properly - quite rightly paranoid that no one was reading his newsletter); in my new job I'm not currently responsible for our e-news. However... I'm still so interested! I think it's important to remember the 'people' factor: PEOPLE open emails; give them something to open: teaser articles that require clicking through to read the rest; graphics that BEG to be seen (charts, stats, etc) with text tags saying "if you can't see this image..."; and stuff (like this article!) that asks for a response - us PEOPLE like to feel like participants, not just passive recipients.


Aug 15, 2006 - Kevin Garber of Melon Media www.melon.com.au says:
I totally agree with this article. It is almost a daily discussion with one of our clients not to take the read rates as literal read rates, but rather use them to compare them campaign to campaign. I also confidently tell clients that read rates can be considered a minimum as there will be people reading with images off etc that won't be counted. At least this counts in their favour. Another trick to increase confirmed read rates is to link every image in the email to a url as recipients seem to want to click on anything. Our system then can confirm that the email was read even if the images were switched off. Kevin


Aug 15, 2006 - YF Juan of Accellion says:
Hate to be a contrarian in this discussion. But, I think there is an unwarranted emphasis on open rate. Sure, it is a statistic that makes the work look more scientific. But, like the famous saying goes, "there are lies, damned lies, and statistics." Instead of measuring open rate, it seems to make more sense to measure click-through's. And, for everything that open rate is purported to help determine such as A-B testing, I do not see why click-through's would be less valid of a measure. I know what I am proposing is a bit on the extreme side, but this is a lesson I've learnt from SEM - "conventional wisdom means nothing and you have to show me the money..." Warm regrds, YFJ


Aug 15, 2006 - Katie of Signal-Tech says:
I read all my e-mail through preview panes - so I would always count them as an open. . . I'm probably weird for that, right?


Aug 16, 2006 - KK of Association says:
One non-scientific (it must be because I work for a scientific association) suggestion I have to measure "general success" is to put a quick reader survey at the end of your e-mail with a splash page showing the result. Another trick that has worked with our organization of brainiac members is to offer a trivia question or game where they need to respond. Winners and results are annouced in the next issue/e-mail, etc. This has been extremely popular with our membership. Our organization uses Outlook and I do use the preview pane. Its definitely an effective way to manage your e-mail without letting it consume your day. Any e-mail I view in this preview mode, I would consider to be opened. Just another constraint marketers need to deal with. Good thoughts, Anne! Very helpful and insightful as always.


Aug 16, 2006 - Leah Barber of www.performancebike.com says:
Ophir has open rates right on... open rates mean nothing unless viewed in comparison either to your baseline or against a test group. So what if our open rate is above average for your industry? As email marketers, we get paid to exceed what is average at our own companies, and that applies to open-rates, click-throughs, AND conversion.


Aug 17, 2006 - Ophir Prusak of Resperate.com says:
1 - I see my previous posting was cut off. If there is a posint limit - you should mention on the form. 2 - You REALLY need to keep some of the formating when posing comments. 3 - Especially when entering list items like this :)


Aug 22, 2006 - Jason Henderson of Big Marketing says:
Don't the big ESP's have a feature where they count a click as an open? My in-house program does. That way someone reading the plain-text version can still be counted as an open if they click to the site. Very useful.



Post a Comment

Note: Comments are lightly moderated. We post all comments without editing as long as they
(a) relate to the topic at hand,
(b) do not contain offensive content, and
(c) are not overt sales pitches for your company's own products/services.










To help us prevent spam, please type the numbers
(including dashes) you see in the image below.*

Invalid entry - please re-enter




*Please Note: Your comment will not appear immediately --
article comments are approved by a moderator.

Improve your marketing

Join our thousands of weekly Case Study readers. Enter your email address below to receive MarketingSherpa news, updates, and promotions:
Note: Already a subscriber? Want to add a subscription?
Click Here to Manage Subscriptions