How Multivariate Test on Banner Ads Boosted Clicks 50.4%

CHALLENGE
Tests that improve creative are *the* most effective online ad tests, according to data from MarketingSherpa’s new 2008 Online Advertising Handbook & Benchmarks. Creative issues were the dilemma that Kevin Kohlmeyer, Emarketing Specialist, Unicel, part of Rural Cellular Corp., faced when it came to optimizing banner advertising across a network.

Kohlmeyer and his team had tried several different copy combinations but didn’t know which worked best. Nor did they know which phone product lured the most consumers to click through and learn more about their services package. Plus, they didn’t know which banner size was optimal.

“For these types of things, we had been looking at industry research to find what certain attributes existed that could be important to consumer prospects,” Kohlmeyer says. “But, we needed to take some steps to verify what really worked for us in particular.”

CAMPAIGN
Kohlmeyer and his team found out “what works” and “what doesn’t work” for Flash banner ads by running an elaborate multivariate test over four weeks. The test included 36 different possible content combinations.

In the end, the goal was to put together a *super banner* with the top-performing creative elements for the campaign. The approach was a stark departure from what they had done in the past with Flash banner advertising.

“We’d put out something like a handset offer or a service-plan promotion across a network,” he says. “A lot of times, it tended to be something limited and something that was consistent with our print ads. But, it was time to do more online.”

The multivariate test showed Kohlmeyer what three key variables, in particular, drive their ads, and what worked best:

-> Key variable #1. Copy

Kohlmeyer and his team tested four lines of copy. Each was attached to an ad with the picture of a cell phone being offered with the service package.

The copy they tested:
o “Buy online and get free shipping on your order”
o “Buy online and we’ll waive your activation fee”
o “Buy online and get free shipping plus no activation fee (over a $30 value)!”
o “Start Shopping Now!”

-> Key variable #2. Product offered

Kohlmeyer knew that pitching consumers with the right name brand can make a big difference. “We looked at what products consumers were selecting once they were already to the site. It made sense that they would bring in prospects as well.”

So, they tested three popular, low-to-medium-cost phones that already were working on their ecommerce site:
o Motorola v197
o Motorola Razr
o Sony Ericsson w200

-> Key variable #3. Banner sizes

Kohlmeyer’s team also knew that using the right banner size could dramatically improve clickthroughs and sales. But which size? (See creative samples below for the winner.)

Here are the three significantly different pixel combinations they included in the multivariate test:
o 300x250 (box ad/often called a “big” ad)
o 160x600 (skyscraper)
o 728x90 (leaderboard)
RESULTS
The testing and numbers crunching paid off big-time for Kohlmeyer and his team. The data generated by the test was used to build the super banner that saw a 50.4% increase in clickthroughs in their next campaign and a “big win” on the sales side, although Kohlmeyer declined to be more specific.

“We’ve seen a healthy overall tick in sales,” he says. “A lot of our sales -- originating from banners -- come over the phone and via other channels. Banner advertising is a significant traffic driver for our multichannel efforts. And this test has helped us improve along those lines.”

Here are the individual test results for each variable as well:
- “Start Shopping Now” copy won out over “Buy online and we’ll waive your activation fee” by 12.2% in clickthroughs.
- “Start Shopping Now” copy got 16.6% more clickthroughs than “Buy online and get free shipping on your order.”

“Getting rid of the ‘loser content’ did help produce a [response] spike. And, what the testing also did was validate some hunches we had or reconfirm what we had seen in the past,” Kohlmeyer says. “At the same time, the fact that ‘Start Shopping Now’ was the best performer was a surprise to us. In the past, we had assumed shipping costs and activation fees were a barrier to purchase and emphasized the fact they were free online. This appears to have been an incorrect assumption. So, we’ll start downplaying it in the future.”

As for which product won:
- Motorola v197 beat the Motorola RAZR by 4.5%.
- Motorola v197 beat the Sony Ericsson by 12.2%.

Kohlmeyer says the margins were enough to feature the v197 in their next banner ad. “It has been a staple in our lineup for a while and has always been a strong seller. It was helpful to know that what sells well is also effective at driving traffic.”

The big winner in the banner size proved that bigger isn’t always better:
- The 300x250 banner (box ad) beat the second-place 160x600 banner (skyscraper) by 27.3%.
- The 300x250 banner trounced the 728x90 (leaderboard) by an even bigger margin.

Data from Sherpa's new Online Advertising Handbook supports this finding. According to a January 2008 survey of marketers by AdJungle for MarketingSherpa, 300x250 ads get the highest clickthrough rate: 62.2% more than skyscraper ads and 27% more than leaderboard ads.

Kohlmeyer says that they had been predominantly using larger pixel sizes like the skyscraper and leaderboard. “We saw that we had to do some downsizing.”

Useful links related to this article

Creative samples from Unicel’s banner ad test:
http://www.marketingsherpa.com/cs/unicel/study.html

DesignBlox - technology provider that made the test possible:
http://www.designblox.com/site/

Kelliher Samets Volk - Unicel’s marketing agency of record:
http://www.ksvc.com

Specific Media Inc. - banner ad network employed on the test:
http://www.specificmedia.com/

Unicel:
http://www.unicel.com