CHALLENGE

Andy Wolf, Director of Ecommerce, Tea Forte, and his team considered trying a new tactic to introduce a new product line. Up to that point, all product introductions had been done exclusively through email.

They mulled launching the new product on their homepage particularly to see if it would have an impact on *overall* conversions for all their product lines. But first they went through some intense internal debate on whether an introductory product deserved homepage ‘real estate’.

“The introductory product may be a good one,” Wolf says. “But the problem can be that it isn’t a core product.”

The debate ended with a decision that they simply had to run some tests to determine the impact of introducing new products on their homepage.

CAMPAIGN

Wolf and his team actually committed to running two tests as they launched their ‘Sanctuary’ line of teas. Here are the steps they took:

Step #1. Create separate A/B tests for product line

They first developed two separate A/B tests to modestly promote the new teas. Both tests ran for about four weeks, during which the Sanctuary brand appeared near the bottom of the homepage for the test group. It didn’t appear there for the control group.

“We were looking to tease out the interactive effects the product launch might or might not have on overall site conversions.”

The first A/B test focused on email recipients. They wanted to find out how an email message could affect their visits to the website. They evaluated two segments: those that opened the email vs. those that didn’t. Subscribers that opened the email and clicked through were taken to the product details page for the launched items. The ones that made a purchase during the test period were compared against the recipients that didn't open the email at all.

For the second A/B test, they evaluated people who came to the website via search engines or by keying in the URL. For the non-email recipients who logged on, half saw the new product on the homepage and the other half did not. They were assessed to see if they made a purchase during the test period.

Step #2. Employ a big sample for email

For the email part of the study, they sent the message to their entire subscriber list because they wanted to test against a big sample. The message used Wolf’s normal template, including their blue ‘SHOP’ button in the middle of a large product image. Once again, clickthroughs were sent to a product details page.

"Our email campaigns direct people to the product page. This is pretty typical; it makes the click seem more relevant by immediately providing more information and a buy opportunity on the featured products. The test on the homepage was run in parallel with the email campaign in order to try to measure the impact of the repeated messaging."


RESULTS
What they discovered surprised them while allowing them to optimize their product launch approach: "I got dramatically different results on the test [groups] vs. the controls."
Perhaps most surprisingly, they learned that including the product launch on the homepage can alter the conversions on their website as a whole – although the effect showed itself in different ways.

“That the overall result showed a cumulative effect from repeated messaging was not surprising. What we found surprising was the strength of the effect, despite a relatively minor creative change."
Key conversion numbers (any purchase) from the test:

- People who opened the email and saw the test creative (new product on homepage) converted 17.2% better than the control group.

- For those who did not receive an email but still saw the creative, conversions dropped 24.3% compared to the control group.

“The interesting thing is -- in the past, I probably would have launched the product in the email and shown it on the homepage for *everyone* visiting the site. What these results show is that I should use that strategy only for the people who opened the email. I need to put the new product up on the homepage for them, but not for the rest of the universe.”

Here are the two big lessons Wolf and his team learned from the test:

Lesson #1:
Never underestimate the importance of messaging and design, especially when doing product launches.

“It taught me that product introductions require multiple touchpoints in order to sink in. And, it may be more valuable to repeat messaging rather than the frequent temptation to try to get as many different merchandising stories as possible into a site.”

Lesson #2:
All hail segmentation.

“As a marketer, I shouldn’t be surprised that segments matter, nor that sequenced messaging matters, but the strength of the combination was surprising. For certain, we will be looking at ways to reinforce email messaging on the website going forward, particularly now that we can limit messaging changes just to the people that open each email.”

Useful links related to this article

Creative Samples for Tea Forte:
http://www.marketingsherpa.com/cs/tea_forte/study.html

Homepage Offer vs. Product Landing Page - Search Test Results
https://www.marketingsherpa.com/barrier.html?ident=3065

Before & After: 4 Steps to Identify Best Redesign With Multivariate Tests & Lift Lead Gen 85.3%
https://www.marketingsherpa.com/barrier.html?ident=3030

Knotice, Ltd. – provided the backend marketing system:
http://www.knotice.com

Tea Forte:
http://www.teaforte.com/