As Hewlett Packard's Director of Worldwide eBusiness Research & Metrics, Seth Romanow oversees the gathering and analysis of four terabytes of data a year.

To put this in perspective, a single terabyte is approximately a thousand billion bytes. It is a heck of a lot of data.

Over the past twelve months Romanow has used this data to improve HP's site design so much that they doubled its visitor-to-buyer conversion rate. (Yes, doubled.) Here are some specifics on how they achieved this remarkable result:

-> Basics on Romanow's Analytics Group

First of all, before HP began collecting customer data they brought in an independent auditor, the site is certified by BBB online, and the privacy policy posted on the home page has an explanation on how to *not* to accept cookies.

In terms of data collection, Romanow says, “Any data we collect is anonymous. One the public side I think that’s where you draw the line on this, we don’t collect any PII so your record is one of 2 terabytes of information. From there we’re very stringent on how the info is used, how it’s secured, and who has access to it within HP.”

HP uses the data it collects to, “Get down to a very granular level on a site and look at it page by page.” Romanow says, “Then if a warning bell goes off we have tools to diagnose what’s going on from a dynamic point of view—to see if there is an issue with that page in terms of dispersion or leakage, high back-tracking, etc.”

Before HP changes information architecture or content they pretest every hypothesis by diverting a segment of visitors to the test site before a change goes live. Then, in weekly meetings, the team tracks changes on the site to determine overall impact.


-> HP’s Three Levels to Measuring Online Experience:

Level #1. Data collection and distribution.

These metrics from traffic and click-stream data focus on site usage and customer behavior. Data like page views, user visits, conversion rates, usability and satisfaction is distributed out in basic reports to HP’s various business dashboards.

Romanow explains, “We’re basically a large data collection group. We compile information, big flags or problem areas appear, and we segment that data out to the right divisions. If we see that a category like conversions is up or down we can look into it more deeply.”

Level #2. Specific campaign analysis

HP then takes the information and looks at it in more detail in terms of how specific campaigns are doing. Using click-path analysis, this deeper dive looks at indications like high backtracking, prospect loss, abandonment, and high and low dispersion rates in terms of how HP might improve the customer experience.

“We’ve developed some models around desired paths based on where we’d like people to go. So if the purpose of the page is to get a visitor to the check-out page and that’s not being achieved then something is wrong with the page.”

It is also important to keep the Web page itself in mind when looking at data like dispersion rate. Romanow highlights an example, “It’s good if the home page has a high dispersion rate, but on the checkout page that’s bad.”

HP also determines how well ads are yielding to purchases. “It’s important to manage internal ads as well as external ads that run on sites like Yahoo, ABC News, and Disney. We provide tools to measure customer satisfaction with demand generation support and doing analysis around clickthroughs.”

This type of in-depth analysis, and proof that it pays off has led HP to predictive modeling.

Level #3. Advanced Analytics and Predictive Modeling.

After playing around with merchandising in different parts of the sale cycle, HP has found that messaging in certain parts of the sales cycle we can affect conversion.

Romanow explains, “If we can understand where somebody comes to the site and provide them with messaging, such as a bundle on sale or a price point a special offer, we hope that being able to somewhat predict that will increase the prospect of sale.”

This high-level of statistical analysis requires extra brainpower. Romanow notes, “We’ve got statisticians and PhD’s working on the statistics side of the house in terms of what we do in analytics and we have very experienced data miners supporting those statisticians.”

HP looks advertising life cycles to predict how long ads work in terms of clicks to the web site as well as conversions. Romanow says optimizing ad placement, “allows us to more efficiently tune the kinds of messages the advertising team is sending out as well as the kinds of banners placed on the site.”

His Group also is currently working on a novel clustering approach to segment visitors, build a predictive model and compute the likelihood of conversion.


-> How HP Used Analytics to Sell $1 Million More iPAQs:

HP was not making as many sales of iPAQ handhelds and related items as it had expected to. Romanow's group sprang into action.

Romanow explains, “When we went in to take a closer look at what was happening with path analysis, we saw there were issues with how the site was constructed.” Survey data told HP there was low satisfaction overall, with especially poor ratings on navigation. Customer comments in the text box backed up those feelings.

HP revised the home page and the product area to be a more comparative grid so people were able to see all the different products on a single page. The options page was also removed. By removing the middle, option detail page, HP streamlined the procedure before checkout.

Romanow notes, “We needed to move customers more quickly towards putting a product in their shopping cart. Now the buy links go directly to the specific item in the store.”

As a result of that simple change, the conversion rate went up 83%, revenue increased 25% on iPAQ options, and HP also saw a yearly gain of a million dollars.


-> Seven Ways HP Tests, Measures, and Compares Online
Performance:

#1. Live panels: Romanow's team recruits both customers and non-customers for a panel to test site tools and potential changes to the site areas.

In a lab setting 10-15 consumers are given a task, and as they click through the site HP monitors their path and asks questions about their level of satisfaction. Also panel members are asked to do comparative tasks on competitors' sites and HP tracks them as they click through.

Romanow explains, “One of our benchmarks is time to task. We look at how long it takes somebody to do something on our site and then we benchmark it with other companies offering similar tasks on their sites."

“This is a great way to engage the brand by asking a visitors opinion of the brand before they came to the site and after.”

#2. Nielsen//NetRatings: HP uses Nielsen//NetRatings to track competitors by traffic. Keynote also measures against the top 40 sites in terms of page rendering times.

#3. Client-side tools: Client side tools to track customer behavior on the site—primarily through click-streams.

#4. Server-side tools: On the server side HP collects log data such as downloads and server performance.

#5. Custom tools: HP has also built their own tools to measure advertising and merchandising effectiveness and calculate ROI.

One such tool, E-Marketing Optimization Methodology (EMOP), helps HP measure the amount of volume that comes in to the site from clicks on ads externally as well as clicks on ads within the site, and tracks them all the way through to conversion. HP also analyzes what is put in the cart and purchased.

#6. Eyetracking equipment: HP has begun testing an eye-tracking machine to see how people review pages and what attracts them to different parts of a page. A camera on a computer monitor is tuned to a persons retina to track eye-movement. Romanow explains, “We can ask someone to download a page and watch how their eye moves throughout the pages and relate that to how they use the site.”

#7. Online surveying program: HP's survey is threaded in several hundred locations throughout the site. HP is sure to use sampling to ensure they get the right balance of responses based on the site, as some areas are higher traffic areas than others.

Romanow explains, “The survey is placed to obtain a certain sample size and to make sure those different samples are valid in different areas of the site.”

Visitors are asked to rate things like, navigation, visual appeal, content quality, up-to-date content, speed of page rendering, and overall site performance on a five point scale. “The focus of the 13 questions is to gauge sub standards in the satisfaction area,” explains Romanow. “But, overall we want to know whether or not they can complete the tasks they came to the site to do.”

He says one of the most helpful features on the survey is the inclusion of a text box for customer comments.

#8. Tracking extranet use. Romanow explains, “We look at customer behavior within our extranet to alert our team to any issues on the extranet in terms of how people are using the site.”


-> Romanow's advice to other marketers

“Using the website, we can measure our business day in and day out,” Romanow says. “But it also goes back to the inherent cost to running a site. We want to ensure that we’re optimizing the tools that we have in the site, whether it’s a store, a high-end configuration, or the home page. If you’re going to invest in an asset like a website you have to make sure that the investment is a good one.”

“Experience shows that small changes over time can yield significant returns in the areas of customer satisfaction, conversion rates and better utilization of product support,” says Romanow.

"It really comes down to using web site and campaign performance optimization strategically where it can make the most difference. The investment, tools and processes will follow."

He adds this happy thought for marketers battling to get more budget for Web analytics, "Over the past 18 months or so, we've seen an interesting trend: the more information we provide, the more business owners use it and, in turn, the more they want.”