CHALLENGE
Every eretailer knows offering free shipping will raise conversions. 


But, can you imagine how much of a difference it would make if you’re marketing extremely large and heavy products online? Vintage Tub & Bath’s average SKU weighs 350 or more pounds and has to be delivered by a trailer.

General Manager Allan Dick thought the free shipping offer was prominent enough throughout the site -- but his team felt differently. "We have been offering free shipping for a number of years, but we wondered if we were promoting it enough."

There's a time for committee meetings and strong opinions in Web site design, and then there's a time for data. Dick decided that time was at hand.

CAMPAIGN
The team decided to run tests via an overlay tool in order to discover the best way to promote shipping offers without having to revamp the whole site or split traffic, which can be cumbersome. And while they were at it, they began to test other design elements they’d long argued about too. Here are test details for two of the most useful ones.

Test #1. Home page images versus text

Vintage Tub & Bath’s default home page was text-heavy, featuring a small product photo on the left that included the words "Free Shipping" next to a large block of text.

Dick's team decided to test something "super splashy." They tried five different creative executions, each featuring a large product “hero” shot that eliminated most of the text. Each product shot included a line of copy. Because there were few variations of the creative executions, it was run as an A/B/C/D/E/F test (the same as an A/B test but with more versions) rather than a multivariate test. These executions were:

a. Default – control using the current home.

b. Large image of a clawfoot tub on a beach, overlaid with the words, "objects of desire."

c. Large image of the tub in the background, with a child wrapped in a towel in the foreground, and the words, "the most precious things."

d. A woman relaxing in a tub surrounded by candles (only head and neck showing), with the words, "indulge yourself."

e. Same image of woman relaxing in tub, but with the words, "hide from your kids."

f. A woman's silhouette, draped in a towel, about to climb into a tub, with the words, "satisfy your dreams."

Each tested image included, at the bottom, the phrase: "Best price. Free Shipping on all orders. Safe and secure ordering."

Test #2. Category page test

The Vintage Tub & Bath category pages had been optimized well for natural search -- but that meant they were text-heavy, and the text was high up on the page. Dick and his team decided to test more tub and bath pictures on their clawfoot tub category page and to give a better brand experience.

They designed a test that took three elements into consideration:

a. The "main" area of the page

In the original clawfoot category page, the top third of the page consisted of the heading, Clawfoot Tubs, followed by a paragraph of dense text. Dick's team tested that default against a new treatment: a large image of a clawfoot tub on the left with shorter, bulleted text just to the right of the tub. On the far right were two thumbnail images under the heading 'Top Selling Clawfoot Tubs'.

b. Link text

Under the main area of the category page were six images (four over two) of products. Each image included a link to the product page. In the default page, the links were in a small, sans serif, underlined blue text. Dick's team tested the default against a new treatment of the link text: a serif font, larger and not underlined.

c. Promotion

Between the main area of the page and the product images below, Dick's team tested the Complimentary Shipping promotion reinforcement, which previously was not found on the category page. They added a blue line across the page. The line was interrupted in the center of the page with the words -- also in blue -- "Free Shipping On Every Order!"

The three new elements plus the default could have been put together in eight different ways, or "recipes." Using multivariate testing, Dick's team needed to test only four of the eight possible combinations to find a winning version of the category page. They were also able to determine which element of each page contributed the most to conversions.



RESULTS
"We did have a clear Home Page loser," Dick says. "We thought the woman wrapped in a towel by a tub, with the words "satisfy your dreams" would give an image of calm and quiet, but that one was a clear loser." Three other findings:

--The image of the tub on the beach ("objects of desire") distinctly beat the image of the child wrapped in the towel ("most precious things").

--The phrase "indulge yourself" drastically beat the phrase "hide from your kids" on the image of the woman relaxing. "We thought the 'hide from your kids' line was funny, so we put it up there. It was more like a joke, but it didn't harm conversions," Dick says.

--The default page was jockeying for position with the "indulge yourself" (woman in tub) and the "objects of desire" (tub on beach) on a daily basis.

For future home page tests, Dick and his team believe that, by combining some of the branded treatments (big images of tub) with the text elements (new link treatments, calling forward the free shipping), they can beat the default by a lot.

"It's valuable information, but it's not always obvious what's next," Dick says. "You refine, move forward and learn something. We're going to be testing like fiends next year."

As for the category page tests: "I was surprised how well the free shipping offer did once we began banging it forward," Dick says. "We saw a huge increase in conversion rates as well as average order size."

In fact, the increase in conversions and average order size could translate to approximately $2 million in increased revenues next year. The free shipping element of the category page tests had the biggest impact on conversions, followed by the link copy treatment (underlined treatment worked best).

After the test had run for two weeks -- reaching an estimated 97% confidence level -- his team halted the test, and then ran it again. The results were verified, though they weren't quite as spectacular. "It went from outstanding to great," Dick says.

Useful links related to this article

Creative samples from both sets of tests: http://www.marketingsherpa.com/vintub/study.html

Offermatica -- the testing software Vintage Tub & Bath used to conduct multivariate and A/B tests with overlays easily without splitting traffic: http://www.offermatica.com

Vintage Tub & Bath http://www.vintagetub.com

Note: Vintage Tub & Bath is a member of Shop.org, a forum for retailing online executives to share information, lessons learned, new perspectives, insights and intelligence. More info at http://www.shop.org