by David Kirkpatrick, Reporter

CHALLENGE

Some webpages have a very clear conversion goal -- purchase, registration, an opt-in, or some other action that directly leads to a lead capture or a sale.

Other pages are designed with a softer metric in mind – deeper engagement with your content, to simply send visitors from a home page to desired information, or for branding. In that case, what you want to see is clickthroughs that take a visitor deeper into the online material they want to find.

One of IBM's major branding initiatives, "Smarter Planet," is located under the company's corporate marketing umbrella and includes around 120 webpages of high-level, thought leadership content. Smarter Planet, is more than a clever branding term; it’s quite descriptive as well, since the initiative is deployed in 73 countries.

"One of the things we wanted to do was increase engagement and clickthrough rates with the pages," said Joan Renner, Content Manager, Corporate Marketing Digital Initiatives, IBM. "We wanted to test out some navigational elements that would help us improve that, and support existing advertising (around Smarter Planet)."

This case study takes a look at the process and methodology IBM used to test engagement of Smarter Planet topics by increasing topic visibility and moving navigational elements on the initiative's website. Renner will also present an in-depth look at this test at the upcoming MarketingSherpa B2B Summit in San Francisco, October 24-25.


CAMPAIGN

The Smarter Planet initiative may is global, so there is a top-down branding element for the webpages in every market. At the same time, the content is localized. Each local marketing team can choose which of the 27 Smarter Planet topics it would like to focus on.

This test was designed to find what combination of topic navigation elements and presentation produced the most clickthroughs, and was conducted on the U.S. and German versions of Smarter Planet.

Renner said Germany was chosen as a test market because of a strong working relationship and the knowledge it would be relatively easy to implement the online testing with the German team.


Step #1. Utilize internal sources to define the test

The opening stage of the test involved two elements: documented marketing and communication strategies, and stakeholder interviews.
Renner stated, "We have well-documented marketing and communication strategies which we use to inform us on what we needed to focus on."

These strategies are documented across IBM by the executive team so they provide Marketing with both a starting point in determining the success metrics for the website (and by extension, the site test), and standards to refer back to when delivering results.

The stakeholder interviews went up to the VP level and were designed to prompt thinking about each stakeholders audience of the site and what content and features serve those different audiences.

A number of areas within IBM were involved in the stakeholder interviews:

o Marketing and Sales Smarter Planet subject matter experts

o Smarter Planet technical team o Industry experts (Smarter Planet content is based around different industries)

o Corporate Marketing team

o Digital Laboratory (IBM's braintrust for Web presence thought leadership)

o Design

o Advertising

o Web managing editors from the worldwide team

o SEO and metrics team

o Digital Social Business program team

o UE/UCD team (addresses site usability)


Step#2. Determine test benchmarks

IBM has a metrics analyst with a team that pulls data on a weekly and monthly basis on all its webpages. The reports generated provide both a high-level corporate view as well as a functional perspective that looks at what is working and what is not working.

Renner said these reports uncovered possible issues with navigational elements. There were drop-down menus at the top of pages and navigation at the bottom of pages, but nothing in the middle of the pages.

"After looking at the data for a while, we said, 'Well, you know, we are ready to try something out,'" Renner explained. "We have the ability to do this."

The reason navigational elements were so important to Smarter Planet webpages was the goal of the page was visitor engagement -- getting visitors to the next click, the next step and deeper into the content. There were no other conversion goals aside from user engagement with the online material.

The site traffic reports provided IBM with two main insights:

o Traffic to the footer topics is generally higher than headline topics, indicating an opportunity to increase traffic to the header topics.

o Smarter Planet Homepage is the highest clicked topic in the header. And if a user was on the homepage, click on the Smarter Planet Homepage logo simply refreshed the page.

IBM's takeaway from the second insight was page visitors wanted to learn more about the initiative and likely thought clicking on the logo would lead to additional information. The solution was to remove the logo and make the Smarter Planet "about or overview" link more prominent.

To prepare for the test, IBM developed groups of metrics to track and segments to target:

Metrics

- Primary metric

o Aggregate topic navigational icon clicks (on icons found at the top and bottom of the page)

- Secondary metrics (some of these elements are only found on specific test treatments)

o Top (header mid-page ribbon topic bar) topic clicks

o Header topic clicks

o Bottom topic clicks

o "Top 10" topics clicks (top ten topics determined by user and IBM interest replaced alphabetized list on one treatment)

o Overall clicks

Segments

o Referrer: Google, Yahoo, Bing!
o Visitor stage: new to IBM's testing solution
o Visitor stage: repeat to IBM's testing solution
o Time of week: weekday, weekend
o Referrer: paid search, natural search, direct, IBM.com


Step #3. Use "agile" and "Scrum" development methodologies

"Agile" and "Scrum" are interrelated product development and project management concepts more closely tied to software development, but can be applied to other business areas. The hallmark of both is working in short time frames, such as two-week iterations, and continually making incremental changes instead of fully developing a large plan of action before deployment.

Renner said IBM has been using agile and Scrum for about a year, and a big advantage is when tackling a large project, such as evaluating the performance of a webpage, the overall effort can be broken down into smaller pieces so if things start moving in the wrong direction, that process can be stopped and refined.

In looking at the Smarter Planet site, IBM found ten areas to test for possible change and the topic navigation test ranked high on the list.

In the agile development world, the entire group of test areas are considered an "epic" and the specific top navigation test is called a "story." The idea at IBM is a "story" can be completed within around two weeks with deliverable results at the end of that timeframe. Each story helps build on the "epic."

Once top navigation was identified as a high-priority item and test benchmarks were determined using site statistics and internal sources, the "story" for the test was created:

The template, "As a I would like to so that ," became: "As a site visitor, I want to quickly see my industry or technology area so I can quickly find the content I want."


Step #4. Run the test

IBM partners with a vendor for its online testing, and the top navigation test was divided into four elements -- the control and three treatments (called "recipes" in IBM's nomenclature) -- and was conducted on the U.S. (also considered the Smarter Planet worldwide site) and German Smarter Planet websites. The A/B/C/D split test ran for around two weeks on 100% of the site visitors.

Renner explained the test, "We saw if we moved some of the navigational elements around on the page, we could actually improve our engagement rates overall and get people around the path to the topic that may be of interest to them."

Pages tested:

- Control (Recipe A)

The topic icons were the key navigational element tested, and in the control appeared at the bottom of the page in an alphabetically-listed ribbon.

- Treatment 1 (Recipe B)

The key change for this treatment was topic icons from the bottom of the page were also added above the fold between, "This week's feature" and "Smarter development" as a mid-page ribbon.

- Treatment 2 (Recipe C)

In this treatment, the below-the-fold icons remained the same (topics listed alphabetically), and the above-the-fold ribbon icons changed to the top ten topics.

- Treatment 3 (Recipe D)

This treatment was a radical departure from the first two. Above-the-fold topic icons were removed in favor of a drop-down menu listing all topics. An icon next to the drop down menu changes as each topic is hovered over with the cursor.



RESULTS

Treatments 1 and 2 outperformed both the control and Treatment 3. All results metrics are at a 99.9% confidence level.

Treatment 1:

o Increased aggregate topics clickthroughs 35.15%

o Increased clickthroughs to header and mid-page ribbon bar topics 109.13%

o Decreased clickthroughs to bottom ribbon topics 64.29%

Treatment 2:

o Increased top 10 topics clickthroughs 128.76%

Treatment 3:

o Increased all clickthroughs, excluding topics, 31.38%

Because of the strong clickthrough to top 10 topics (instead of alphabetized list) for Treatment 2, IBM determined it was the "winner," and IBM's testing system allowed the winning page to be immediately implemented across the website.

Other takeaways from the test include a list of potential follow-up tests including:

o Remove the drop-down menu at the top of the fold

o Re-prioritize the top 10 topics

o Reduce the amount of center copy

o Reduce the amount of videos within the center copy

o Promote the top ribbon above the hero image

o Test using only the mid-page ribbon (remove the footer ribbon)

o Redirect users that click on the Smarter Planet logo to a random topic

o Add the mid-page topic ribbon bar to topic pages

Another key takeaway from this test that produced significant clickthrough gains is explained by Renner, "We haven't spent a whole lot of money and a whole lot of time."

She added, "Going beyond the metric analysis on clickthroughs, you are getting this user behavior information. We can offer our visitors a much richer, more fulfilling experience by understanding what they need and want, where they are going, and providing that to them."

Joan Renner will present an in-depth look at this test at the upcoming MarketingSherpa B2B Summit in San Francisco, October 24-25.


Useful links related to this article

CREATIVE SAMPLES:
1. Tested icons
2. Control
3. Treatment 1
4. Treatment 2
5. Treatment 3

IBM Smarter Planet

Members Library -- Master the Art of Multivariate Testing: 7 Lessons from Avis Budget Group

Members Library -- Website Redesign: Customer surveys and testing help increase conversion by 21.6%

Members Library -- How to Plan Landing Page Tests: 6 Steps to Guide Your Process

Quick Lift Ideas: How switching a headline with a sub-headline could increase conversion (+6 more ideas)

Website Redesign: Wondering what to test? Just ask your customers

Landing Page Optimization: 2 charts describing the best page elements to test and how to test them

Homepage Optimization: No single metric will do