Formalizing a testing and optimization program and really taking command of data analytics can be a powerful marketing asset. Four years in the making, this case study on SAP's Test Lab is so extensive, we broke it down into two parts.
This week covers the founding of the program, the overarching strategy behind the Test Lab and aspects of its operational process. Next week, we’ll dig deeper into some of the challenges the team at SAP faced with the Test Lab, and how this initiative led to "definitive insights" for SAP's marketing.
As a sponsor, I wanted to see the more interesting stuff, and for me and for the entire team that supported Web analytics at SAP, that always meant more efficient marketing, more productive marketing, more successful marketing, marketing that drove the business of SAP.
And, the assumption of course, is that Web analytics — [the magic] happens almost by default. Because, you know as the story goes, once everybody has transparent reporting, once everybody has access to a single source of the truth, once everybody can see what's working and what’s not [working], by default we’re going to start doing things.
We're going to do more of the things that work well, that are driving engagement and getting high traffic, and conversion and consumption rates, and we're going to do less of the things that are obviously and transparently irrelevant because our customers just aren't consuming them.
You're waiting for the change to happen in the way we go to market, the types of content that we offer on the website, the way that content is built and packaged.
You're waiting for the change to occur because the single source of the truth is the original business requirement, but guess what? The reports only get you so far.
Once you got your baselines in place and you know what's good and what's bad, great, but now we have to take it to the next level and do better marketing as a result.
What happens at SAP probably is not unfamiliar to what happens at a lot of places. The change in marketing didn’t occur.
Now, what do I mean by that? Our implementation was wonderfully successful, great reports, great transparency, and we immediately went to a high number of users.
The challenge was, as we talked to a bunch of the marketing folks that were using the Web analytics platform, it really was an afterthought, and not in a bad way.
The reality is that marketing folks are busy, and at SAP, they all have defined jobs, and they're either making great ads or doing customer references, or hosting incredible events — on and on it goes.
If you make Web analytics everybody’s job, you quickly realize it's nobody's job, because nobody sits around on Friday afternoon, at the end of the week, and runs Web analytics reports.
The year is 2013 and the digital platforms are allowing us to test in real-time. Not only is the test itself driving more value, because for the percentage of your users that saw the winning version of the test they're converting higher, but you can deploy the results of the test in real-time as well.
A/B is short term until you have a winner, and then you just turn off "A" and you turn on "B," and it now becomes your de facto Web experience for everybody after the fact. So, deployment becomes instantaneous and you’re getting better results. You know just from a time perspective, that changes almost everything traditional marketers know about testing.
Why is that so important?
Because, it forces the rest of the company to queue and request to be part of the test, and it has to make a business case.
I'll give you an example. If somebody is in a queue and says, "I want to test to get people at a breakfast event in Brazil and I hope to fill up a room with 20 people that can listen to SAP for breakfast," and somebody else comes and says, "I want to test my new shopping cart for the Web analytics — the shrink-wrap software we sell over e-commerce — and I'm hoping this can generate an incremental $2 million in revenue."
You can immediately see where I'm going with our queuing process. We're going to put testing resources behind revenue generation e-commerce as opposed to the breakfast event that hopes to get 20 people to attend a physical event.
The number one takeaway is very simple.
As hopeful and aspirational as we were, or idealistic as we were when we bought into the power of what we were setting up … OK, so we had some aspirations.
We said, we're going to take a leap of faith. We put the platform in place. Then, we went further and put people in place, and I personally went on the line to hire people in lieu of search marketers, email marketers, etc.
I hired test lab people.
We went out and put a new head count in place, and then put the whole process in place for marketing. So, we're out on a limb because we completely believed in what was possible, and we’re now on year three and we still don’t think we've scratched the full surface on how much value there is in this process.
Case Study: See the before and after of an entrepreneur’s successful website redesign
Learn the reason behind the changes and discover the transformative impact from using a rigorous process to get there.Get the Details >
The MECLABS Value Proposition Toolbox
Download these 4 tools to help you evaluate and craft your value proposition.Download the Toolbox >
Case Study: How Aetna’s HealthSpire startup generated 638% more leads for its call center
Learn landing page and call center optimization tips from this successful startup within Aetna.Watch the Video >
Create a Sustainable Competitive Advantage
See how customer-first science transforms the way companies achieve a superior position.Watch the Video >
Free 22-page guide that shows you how to better serve your customers
Get these tactical tools and detailed examples to help optimize your campaigns.Download the Guide >
Receive the latest case studies and data on email, lead gen, and social media along with MarketingSherpa updates and promotions.