MarketingSherpa Video Archive

Testing and Optimization: How multiple testing approaches increased a B2B’s banner ad clickthrough 956%

Cindy Lu, Senior Marketing Manager of Digital Marketing Strategy, VMware



When a company offers complex product lines, it can be a challenge for marketers to distinguish similar sounding products, services and packages from one another. This awareness hurdle was the challenge the marketing team at VMware faced.

"We have a lot of products that are in different stages of maturation," Cindy Lu, Senior Marketing Manager of Digital Marketing Strategy, VMware, said. "We have some products that are emerging that are still beta products. We have some products that are getting more traction, and we have some products that are flagship products."

At MarketingSherpa MarketingExperiments Web Optimization Summit 2014, Lu spoke about the differentiation challenges that come with having a varied product line, the discoveries she and her team made through optimizing VMware’s website and conversion funnel, the importance of testing quick wins and the value of running multiple tests.

VMware provides cloud and virtualization software and services, and, according to Lu, the B2B’s many product offerings pose a challenge when it comes to marketing.

"Unless you really know what you’re looking for, it’s kind of hard to find [it]," Lu said. "Part of our digital strategy at VMware is helping each prospect find what they’re looking for, and we actually talk about the conversion funnel and how to measure that."

Through testing VMware’s website and conversion funnel, Lu was able to increase the clickthrough rate on a product’s CTA by 38% and increase clickthrough for a hero banner ad by 956%. Learn the five most important optimization discoveries Lu made in her optimization efforts.

Step #1. Optimize content for the customer

Lu’s testing journey for VMware began when she and her team noticed a page anomaly. Specifically, they noticed the exit rate for the vCenter Log Insight was 20% greater than it was for any other product page.

The team developed a hypothesis to determine if there was an aspect of this product’s overview page that was not performing well for its particular customer segment. A split test was then implemented with 50% of the page’s traffic landing on the overview page and the other 50% being redirected to the vCenter Log Insight’s features page.

This redirect to the product’s features page increased clickthrough for the page’s "Buy Online" CTA by 38%.

"I guess the key takeaway there is, by doing this test, you kind of understand that your prospects need the right amount of information before they commit to taking that action on your webpage," Lu said.

Step #2. Rapidly iterate and test

After testing helped resolve the vCenter Log Insight’s exit rates, Lu and her team decided to utilize their success to promote more testing.

"Businesses move pretty fast, so one of the things you really want to do is capitalize on the momentum that you have when you’re testing," Lu said.

Following up on the team’s previous test, VMware ran a second hypothesis, which was focused on treating its prospects more like consumers instead of like enterprise users. This involved testing a new template and design for the overview page.

"We were trying to isolate what wasn’t working for that particular set of customers," she said.

This second test with the new consumer layout resulted in a 14% decrease in clickthrough. Lu attributed this decrease to the company’s understanding of VMware’s customers.

"Fundamentally, what we learned was our hypothesis was wrong and that these users really truly are enterprise users and that switching over to the consumer template didn’t work," she said. "You really need to understand your customers and what they’re trying to find on your website."

Step #3. Identify and capitalize on quick wins

When asked which of the five takeaways she has found to be most valuable, Lu focused on the importance of quick wins.

"For each company that I’ve actually worked at, I’ve had the opportunity to quickly identify some of the low-hanging fruit opportunities, and it generally blows everyone away in your organization because they didn’t realize such small tweaks could have such meaningful impacts on actual conversions," Lu said.

By making two small changes to VMware’s Small Medium Business hero banner, the team was able to increase clickthrough 956% for Variant B and 330% for Variant C.

Lu advised, "Make sure you focus on your quick wins and your low-hanging fruit opportunities as you optimize your website."

Step #4. Increase engagement with product differentiation

A major challenge VMware had to overcome was product differentiation. For several products on VMware’s site, distinctions between products packages were not clearly made to users.

"Since the products are of a highly technical nature, we need to make sure that the messaging — the value proposition — is speaking to a technical audience," Lu explained. "They really don’t like what they perceive to be marketing language."

To address this concern, VMware tested advertising the vSphere product line using its hero banner ads again. This time, the control was tested against the bare bones vSphere Essentials (Variant B) and the upsell of vSphere Essentials with the Operations Management Suite (Variant C).

This test resulted in an 86% increase in clickthrough for the vSphere Essentials.

"One of the key takeaways from this particular test is that you really need to communicate to your prospects in the language that makes sense to them," she said. "In our case, it’s a very highly technical crowd, so we really needed to address the technical benefits of vSphere with Operations Management."

Step #5. Tie optimization efforts to the bottom line

Finally, one of the most important lessons that Lu took away from her testing efforts was that the customer journey is constantly evolving. Because of this, marketers’ testing and optimization strategies must evolve as well.

"It’s really essential to understand what your customers are looking for, how it ties into your bottom line and adapt your test to be able to measure that," Lu said. "Actually being able to understand where [your customers] are in the buyer’s journey, where they are in the conversion funnel, will make your messaging and testing much more effective as you continue to optimize your websites."

Related Resources

Reserve your seat at MarketingSherpa Summit 2016 — At the Bellagio in Las Vegas, February 22-24

Web Optimization Summit 2014 Wrap-up: Top 5 takeaways to improve your testing and optimization

Marketing Mashup: Top takeaways from Web Optimization Summit 2014 [MarketingSherpa webinar]

Video Marketing: How VMware's Google Plus Hangout produced a 1,600% ROI

Marketing Research Chart: Where do marketers focus their optimization efforts?