Close
Join 237,000 weekly readers and receive practical marketing advice for FREE.
MarketingSherpa's Case Studies, New Research Data, How-tos, Interviews and Articles

Enter your email below to join thousands of marketers and get FREE weekly newsletters with practical Case Studies, research and training, as well as MarketingSherpa updates and promotions.

 

Please refer to our Privacy Policy and About Us page for contact details.

No thanks, take me to MarketingSherpa

First Name:
Last Name:
Email:
Text HTML
Nov 26, 2008
Case Study

Homepage Product Launch Test Produces Some Surprising Results: 2 Tests

SUMMARY: Optimizing tactics is critical to successful online marketing. Check out how an eretailer tested the idea of including a new line of products on their homepage to see how it would affect *overall* conversion rates.

The two-part test showed surprising results: Conversions took a sizable jump for one segment while the other segment went south. Includes a pair of big lessons learned and creative samples.
CHALLENGE

Andy Wolf, Director of Ecommerce, Tea Forte, and his team considered trying a new tactic to introduce a new product line. Up to that point, all product introductions had been done exclusively through email.

They mulled launching the new product on their homepage particularly to see if it would have an impact on *overall* conversions for all their product lines. But first they went through some intense internal debate on whether an introductory product deserved homepage ‘real estate’.

“The introductory product may be a good one,” Wolf says. “But the problem can be that it isn’t a core product.”

The debate ended with a decision that they simply had to run some tests to determine the impact of introducing new products on their homepage.

CAMPAIGN

Wolf and his team actually committed to running two tests as they launched their ‘Sanctuary’ line of teas. Here are the steps they took:

Step #1. Create separate A/B tests for product line

They first developed two separate A/B tests to modestly promote the new teas. Both tests ran for about four weeks, during which the Sanctuary brand appeared near the bottom of the homepage for the test group. It didn’t appear there for the control group.

“We were looking to tease out the interactive effects the product launch might or might not have on overall site conversions.”

The first A/B test focused on email recipients. They wanted to find out how an email message could affect their visits to the website. They evaluated two segments: those that opened the email vs. those that didn’t. Subscribers that opened the email and clicked through were taken to the product details page for the launched items. The ones that made a purchase during the test period were compared against the recipients that didn't open the email at all.

For the second A/B test, they evaluated people who came to the website via search engines or by keying in the URL. For the non-email recipients who logged on, half saw the new product on the homepage and the other half did not. They were assessed to see if they made a purchase during the test period.

Step #2. Employ a big sample for email

For the email part of the study, they sent the message to their entire subscriber list because they wanted to test against a big sample. The message used Wolf’s normal template, including their blue ‘SHOP’ button in the middle of a large product image. Once again, clickthroughs were sent to a product details page.

"Our email campaigns direct people to the product page. This is pretty typical; it makes the click seem more relevant by immediately providing more information and a buy opportunity on the featured products. The test on the homepage was run in parallel with the email campaign in order to try to measure the impact of the repeated messaging."


RESULTS

What they discovered surprised them while allowing them to optimize their product launch approach: "I got dramatically different results on the test [groups] vs. the controls."
Perhaps most surprisingly, they learned that including the product launch on the homepage can alter the conversions on their website as a whole – although the effect showed itself in different ways.

“That the overall result showed a cumulative effect from repeated messaging was not surprising. What we found surprising was the strength of the effect, despite a relatively minor creative change."
Key conversion numbers (any purchase) from the test:

- People who opened the email and saw the test creative (new product on homepage) converted 17.2% better than the control group.

- For those who did not receive an email but still saw the creative, conversions dropped 24.3% compared to the control group.

“The interesting thing is -- in the past, I probably would have launched the product in the email and shown it on the homepage for *everyone* visiting the site. What these results show is that I should use that strategy only for the people who opened the email. I need to put the new product up on the homepage for them, but not for the rest of the universe.”

Here are the two big lessons Wolf and his team learned from the test:

Lesson #1:
Never underestimate the importance of messaging and design, especially when doing product launches.

“It taught me that product introductions require multiple touchpoints in order to sink in. And, it may be more valuable to repeat messaging rather than the frequent temptation to try to get as many different merchandising stories as possible into a site.”

Lesson #2:
All hail segmentation.

“As a marketer, I shouldn’t be surprised that segments matter, nor that sequenced messaging matters, but the strength of the combination was surprising. For certain, we will be looking at ways to reinforce email messaging on the website going forward, particularly now that we can limit messaging changes just to the people that open each email.”

Useful links related to this article

Creative Samples for Tea Forte:
http://www.marketingsherpa.com/cs/tea_forte/study.html


Homepage Offer vs. Product Landing Page - Search Test Results
https://www.marketingsherpa.com/barrier.html?ident=3065

Before & After: 4 Steps to Identify Best Redesign With Multivariate Tests & Lift Lead Gen 85.3%
https://www.marketingsherpa.com/barrier.html?ident=3030

Knotice, Ltd. – provided the backend marketing system:
http://www.knotice.com


Tea Forte:
http://www.teaforte.com/


See Also:

Comments about this Case Study

Dec 01, 2008 - Doug Carlson of consultant says:
This doesn't seem very surprising. They did an email test where they announced a new product. And then some of those recipients got sent to a home page that showed the product, and some got sent to a home page that didn't show the product. And those who saw the product from the email converted more. Sounds kinda basic. Am I missing something?


Dec 02, 2008 - Chris Heine of MarketingSherpa says:
Thanks for your question, Doug. This test was indeed not run-of-the-mill in its focus, and that is why we found it interesting. None of the email recipients were sent *directly* to the homepage; they were all sent to the products details page that was designated for the new items. The A/B split occurred in that half of the email recipients went to a version of the website where the product launch appeared modestly on the homepage, while the other half got a version of the website where they saw no product launch on the homepage. They were testing the *impact* of including the product launch on the homepage had both for email recipients and regular visitors.


Dec 04, 2008 - Brian Deagan of Knotice says:
Doug...don't disagree. The fact that the top-performing segment was people who opened and the email and saw the homepage creative isn't surprising. Messaging reinforcement works. The part that I found compelling was the tradeoff between telling folks about the new product and selling the old, or legacy products. They would have been better off by nearly 25%, if they simply displayed legacy product homepage creative to this large segment of site visitors, while taking alternative approaches to introduce the new product.



Post a Comment

Note: Comments are lightly moderated. We post all comments without editing as long as they
(a) relate to the topic at hand,
(b) do not contain offensive content, and
(c) are not overt sales pitches for your company's own products/services.










To help us prevent spam, please type the numbers
(including dashes) you see in the image below.*

Invalid entry - please re-enter




*Please Note: Your comment will not appear immediately --
article comments are approved by a moderator.

Improve your marketing

Join our thousands of weekly Case Study readers. Enter your email address below to receive MarketingSherpa news, updates, and promotions:
Note: Already a subscriber? Want to add a subscription?
Click Here to Manage Subscriptions