by David Kirkpatrick, Reporter

CHALLENGE

Testing and optimization can be conducted on many pieces of a marketing campaign, including email subject lines, email content, PPC ad content, Web page layout, landing pages and more. It can also take many different forms: Outsourced to an external testing company; handled internally; conducted with formal rigor including A/B split testing with carefully tracked significance and confidence levels; and even informal "ad hoc" testing that consists of making changes to the marketing piece and looking for a "winning" version.

In the lead-up to Optimization Summit 2011, recent MarketingSherpa case studies have included a detailed overview of Intuit's formal four-test cycle on a high-volume landing page, and a look at WorldAPP's more informal four-stage testing on a microsite that served as its main lead generation engine. Today's case study provides insight into one small business's comprehensive strategic approach to testing and optimization.

How comprehensive? AppSumo, a five-employee firm that provides daily deals on tools for marketers and Web entrepreneurs, has one developer -- 20 percent of its entire workforce -- whose sole company role is to design and execute the firm's ongoing testing and optimization.

CAMPAIGN

AppSumo tests four major areas of its business:

o Email collection
o Email conversion
o Landing page conversion
o Social sharing of its daily deals

And according to "Chief Sumo," Noah Kagan, the company has found two aspects of testing to be effective in optimizing its marketing efforts:

o Pick one area to focus on and test individually
o The "rule of eight"

He explained the rule, "What we've identified is it takes about eight tests and eight optimizations to get one that's going to be really significant to the business."

Step 1. Create a testing and optimization strategy

It's possible to conduct one-off tests on different marketing pieces, say the content in a PPC ad campaign, but mapping out a formal strategy can maximize the effectiveness of testing and optimization. AppSumo's strategy begins with isolating one element to test at a time, and repeatedly testing this element -- the rule of eight -- to find real gains in each area of testing focus.

Kagan described the compounding benefit of executing AppSumo's rule of eight, "Two ten percent improvements a month is a 980 percent increase at the end of the year."
Another aspect of AppSumo's testing strategy is to make major changes in each tested element.

He said, "If you're going to do it (testing), make it dramatic and then pull back. You need significant data to make decisions on (your results)."

He added that testing dramatic changes allows you to see what is making a difference and will point you in the right direction so you can fine-tune the winning test version.

Although AppSumo has built its own internal tool for testing and optimization, Kagan said free, readily available online tools can be effective for smaller businesses looking to test and optimize websites and other marketing efforts.

Step #2. Be persistent in your testing and don't give up

In terms of AppSumo's testing strategy, persistence is key. Kagan said the company conducts continuous cycle testing by choosing one marketing element and running as many as four tests per week on that element. They then return to the element the following week and try to beat those results until a chosen goal or benchmark is reached.

This persistence is also important to give the testing cycle an opportunity to find improvement.

Kagan stated, "I think it's easy to get discouraged and give up when you say, 'I read about this A/B testing article on MarketingSherpa or MarketingExperiments and they had a 5,000 percent increase.' And then you go do it and [the testing] doesn't do anything."

AppSumo has made testing and optimization a key aspect of its staffing decisions. The company is a five-person team with one overall infrastructure developer, and a second developer with the interesting title of "Secret Weapon #2" who focuses solely on testing and optimization. This developer's role is to choose the element to be tested each week and then implement the tests with decision-making autonomy.

Kagan said having a dedicated testing and optimization developer on staff is an effective business move because that developer's decisions directly affect the bottom line. If testing achieves two separate ten percent improvements per month, the position effectively pays for itself.

To continue building on its testing efforts, AppSumo saved every test ever conducted with results, along with what was learned, to refer to in the future and to have that information for any new hires at the company.

Step #3. Engage in testing when it makes business sense

Kagan believes a company needs to exceed a certain threshold of revenue or traffic to engage in testing.

He said, "If you have a thousand users a month [testing] won't impact your business significantly, and you probably have a larger issue of making sure you're solving the right business problem and have the right [solution] set up before these tests actually will do anything for you."

He added AppSumo did not conduct any testing for the first four months of the business, and then began using online analytic tools to implement an early version of the strategy. Later the company brought in the dedicated testing developer who created AppSumo's internal testing tool.

[Editor’s note: Testing with small sample sizes also increases the difficulty in running valid, statistically significant tests.]

Step #4. Don’t fear failure

Kagan admitted continual testing can be daunting because it can take many iterations to find a significant lift and that most of AppSumo's tests fail to see improvement.

But, he added, "It doesn't always happen, and that's part of testing. Being willing to stick with it and keep trying is going to separate the winners from the losers."

When AppSumo adds a losing test to its document recording every test it conducts, along with the outcome the entry includes ideas for future tests to try and improve the results.

Here are examples of both from AppSumo's extensive testing:

o The company tested having its customers name their own price. This did not increase conversion, and Kagan believes this is because it reduced the perceived value of its product.

o The company changed the wording in the body of its daily deal email from "see it" to "buy now." This seemingly minor change resulted in a 36 percent reduction in clickthrough. After a change back to the original wording, clickthrough immediately returned to the previous level.

o A test that improved email collection was asking the website visitor to share the daily deal through social media networks in exchange for an e-book or a download. The company found people were sharing the deal, and then providing an email address to receive the offered benefit.

Kagan said he suggested the sharing call-to-action be presented as a pop-up asking the visitor to "go do this right now." That approach didn't work, but placing the sharing button unobtrusively at the end of the deal's page did result in clicks and eventually email collection.

o AppSumo's best individual result was a pricing test where one deal was offered at $99 and one at $50. The lower deal price resulted in a better than 200 percent increase in conversion.

Step #5. Learn from your tests and continue optimizing

If you are going to engage in testing, take the time to learn from every test -- both winners and losers. Take this knowledge to the next round in the cycle to continue optimizing whatever element of your marketing strategy you are testing.

Email collection is a focus of AppSumo's business. Kagan said, "We found that for our business it's more akin to, 'I don't even care if they buy today. I just want to collect their email so I let them know about upcoming stuff.'" Once the company reached that determination, it decided email collection was going to be the priority.

And as the priority at a business with such a comprehensive testing and optimization program, email collection is a big part of its testing efforts.

In all testing, AppSumo creates a spreadsheet listing its available testing options, possible conversion impact, and time to development. The spreadsheet also includes a score for each testing activity based on a calculation of impact multiplied by the time to develop. These options are sorted by score, and the highest scoring are tested first.

Although AppSumo attempts to run a minimum of two email-based tests every Monday, this group of four tests were staggered over two weeks to ensure they produced statistically significant data. Kagan said the company aims for a 95 percent or higher, confidence level before it finalizes decisions.

He explained, "I've seen so many tests that start with a great win, but end up losing or just end at null. You must wait out to see significance to have better understanding of long-term impact."

Test 1:

What was tested: Incentivize the user on landing page with 20% discount.

Result: Email conversions decreases by 5%.

Kagan's takeaway: "Professional users want to see what they are about to get and don't care about a discount to trade their email address. Give discounts after they know and/or trust your brand or have real incentive to want a discount."


Test 2:

What was tested: Multiple titles for new visitors

Result: No difference among titles

Kagan's takeaway: "The better thing we should do is match titles from certain ads. Also, design elements on the page and page-load matter as much as what you say in the ads."


Test 3:

What was tested: Show email collection call-to-action for all visitors at the website homepage

Result: Increased email collection by 50%

Kagan's takeaway:" We realized that only 0.5 percent of people will actually buy something from our site and only one to two percent will want to actively sign up for our newsletter. So we show the email 'welcome mat' to all visitors. I keep testing removing this, but it does better for email conversion and revenue every-time. People want to see the deal so they give their email to see it, and then subsequently like getting our further emails which they didn't know they'd like."


Test 4:

What was tested: Adding "trust" factors on email collection
Result: 17% decrease in email conversion

Kagan's takeaway: "Don't just assume it works because you read it in a study. Test things to see the real impact. We feel scrolling on welcome pages hurts conversion and there is confusion by having so much going on with our new landing page."


RESULTS

Kagan stated having a target and deadline for testing goals helps keep track of progress and keeps the optimization efforts on track, and not to let a lack of results deter your enthusiasm for testing.

"Look for big, compound, wins," he said. "Big wins are shifts that will dramatically change your business, and complement that with little things that will compound over time. It's a balance of both that we've noticed in our business."

o Results from AppSumo's email collections testing has found email capture increase from 0.5% to more than 5% for all traffic

o With the "name your price" test, conversion increase was negligible and lifetime value of customers was lower than AppSumo's organic traffic

o The social media sharing test led to a 13.5% conversion rate with a viral coefficient of .1 (for every shared deal one person out of ten responded)


Useful links related to this article

CREATIVE SAMPLES

1. "Buy It" email
2. "See It" email button
3. Test 2, version 1
4. Test 2, version 2
5. Test 2, version 3
6. Test 4

AppSumo

Statistical Significance from Professional Research Consultants, Inc. -- Tool used by AppSumo to determine testing confidence levels

Google Website Optimizer -- Tool originally used by AppSumo for testing

Members Library -- Landing Page Testing and Optimization: How Intuit grew revenue per visitor 23%

Members Library -- Landing Page Optimization: How to serve 2 markets with 1 page

Members Library -- How to Plan Landing Page Tests: 6 Steps to Guide Your Process

Landing Page Optimization: Minimizing bounce rate with clarity

Optimization and A/B Testing: Why words matter (for more than just SEO)

Optimization Summit 2011