A highly customizable webpage can deliver hundreds of possible variations, depending on the visitor. That functionality makes it extremely complicated when measuring and testing which version will deliver the best results.

Deepak Nadig, Distinguished Architect, eBay, has grappled with this challenge for several years. His team knows that customized experiences yield better results at the online auction site, and they’ve designed portions of the website to adjust to a visitors’ recent and historic behavior.

But as their pages incorporated more and more variable features, measuring and testing became complicated.

"Measurement becomes very tricky. Traditional metrics like page views are not accurate because you have to define what the page really is," Nadig says. "It becomes evident that I’m not referring to the page at all anymore. I’m referring to a page composed of pieces of content that are organized in various ways."

Complicated or not, a high level of customization often can dramatically improve a site’s performance. Conversions have improved in recent years due to better customization and measurement, Nadig says.

We asked Nadig to describe how his team updated eBay’s website architecture to provide better and more measurable customization -- and to get the rest of the company behind them. Here are five tactics that guided their process.

Tactic #1. Focus on customer goals

Nadig’s team maintained focus on the most important aspect of the website: The customer’s experience. Any changes should move customers closer to achieving their goals, such as purchasing, registering an account, or researching products.

Focusing solely on clicks or impressions is distracting, as they are not a success measurement in themselves. A click is only beneficial if it moves the user toward a useful conversion.

- Qualitative analysis
The team looked at data to make sure that users were progressing to an ultimate conversion. They also conducted usability studies and surveys to ensure that they were meeting customers’ needs.

"We use qualitative approaches like visits where we go to customers’ houses to understand how they interact with the website," Nadig says. "And there is a whole range of other methods, such as eyetracking."

Tactic #2. Rethink webpages and metrics

The team moved beyond the traditional concept of a webpage as a consistent user experience.

Their webpages updated in nearly real-time, with products added, auctions ending, and bid prices changing. Also, each user’s actions dictated a customized selection of content.

In response, team broke pages into their core parts, including:
o Layout
o Modules, or content containers
o Content delivered

Each of these parts had many variations, and resulted in hundreds of possible combinations. The team also had to monitor "partial page updates" when the content in a module changed.

They constructed systems to determine which combination of page elements were working best to bring the users to their end goals. This is a very complicated, tech-heavy enterprise that continues to be worked on today.

Tactic #3. Start small

As the team discussed these concepts they quickly realized that bringing a customized experience to eBay’s entire website would be an enormous task -- especially when dealing with legacy website architecture.

They chose instead to begin working on eBay’s homepage. This experiment would keep the project at a manageable size and avoid disrupting too much of the site at once.

"Traditionally we had the same homepage for all the customer segments and we have been making it more tailored and customized," Nadig says.

- Design test to have an impact

eBay’s homepage is an important part of the website. Picking it as a first target ensured that the experiment would have enough impact convince the rest of the organization that more customization could improve performance.

If they had started with a page that did not provide much benefit, then convincing the rest of the company to expand the project would have been more difficult, Nadig says.

- Choose second targets with similar criteria

After the team had customized the homepage, they moved on the search results page. They chose this page for all the reasons that they started with the homepage: it had a manageable project size and a high potential impact.

Tactic #4. Find company champions

The changes that the team intended to make impacted every department that worked with the website. Nadig’s team needed to find believers in their experiment in each department to help garner support for their plan.

They were able to convince some of these champions of the value of their efforts by showing them the results of their work on the homepage, and later experiments. Then the champions could relate the changes that were needed to their respective departments -- pulling in more support.

- Don’t overinflate expectations

Once other teams noticed how much value could be gained from customization and measurement, they wanted more insights, Nadig says. Other teams’ expectations should be well managed.

"It sort of became this vicious cycle where people asked for more, and you extend it more, and so forth."

Tactic #5. Incorporate legacy technology and habits

eBay is a massive website with a high volume of traffic and transactions. The changes that Nadig’s team made were complicated, and could have backfired, crashed the site, and hurt their users’ experience if they weren’t careful.

Equally important were the expectations of the other eBay teams. Nadig did not want to disrupt their daily routines -- especially when they were pulling for more support. The team had to make sure that all the changes were compatible with eBay’s older systems.

"The business depends on standard reports every day, and we really can’t impact those reports. We have to work our way in a sensible manner."

Useful links related to this article:

Collaborate with Data Anlysts: 4 Strategies to Improve Relationships with IT
http://www.marketingsherpa.com/article.php?ident=31368

eBay
http://www.ebay.com