SUMMARY: Formalizing a testing and optimization program and really taking command of data analytics can be a powerful marketing asset. Four years in the making, this case study on SAP's Test Lab is so extensive, we broke it down into two parts.
This week covers the founding of the program, the overarching strategy behind the Test Lab and aspects of its operational process. Next week, weíll dig deeper into some of the challenges the team at SAP faced with the Test Lab, and how this initiative led to "definitive insights" for SAP's marketing.
As a sponsor, I wanted to see the more interesting stuff, and for me and for the entire team that supported Web analytics at SAP, that always meant more efficient marketing, more productive marketing, more successful marketing, marketing that drove the business of SAP.
And, the assumption of course, is that Web analytics — [the magic] happens almost by default. Because, you know as the story goes, once everybody has transparent reporting, once everybody has access to a single source of the truth, once everybody can see what's working and whatís not [working], by default weíre going to start doing things.
We're going to do more of the things that work well, that are driving engagement and getting high traffic, and conversion and consumption rates, and we're going to do less of the things that are obviously and transparently irrelevant because our customers just aren't consuming them.
You're waiting for the change to happen in the way we go to market, the types of content that we offer on the website, the way that content is built and packaged.
You're waiting for the change to occur because the single source of the truth is the original business requirement, but guess what? The reports only get you so far.
Once you got your baselines in place and you know what's good and what's bad, great, but now we have to take it to the next level and do better marketing as a result.
What happens at SAP probably is not unfamiliar to what happens at a lot of places. The change in marketing didnít occur.
Now, what do I mean by that? Our implementation was wonderfully successful, great reports, great transparency, and we immediately went to a high number of users.
The challenge was, as we talked to a bunch of the marketing folks that were using the Web analytics platform, it really was an afterthought, and not in a bad way.
The reality is that marketing folks are busy, and at SAP, they all have defined jobs, and they're either making great ads or doing customer references, or hosting incredible events — on and on it goes.
If you make Web analytics everybodyís job, you quickly realize it's nobody's job, because nobody sits around on Friday afternoon, at the end of the week, and runs Web analytics reports.
The year is 2013 and the digital platforms are allowing us to test in real-time. Not only is the test itself driving more value, because for the percentage of your users that saw the winning version of the test they're converting higher, but you can deploy the results of the test in real-time as well.
A/B is short term until you have a winner, and then you just turn off "A" and you turn on "B," and it now becomes your de facto Web experience for everybody after the fact. So, deployment becomes instantaneous and youíre getting better results. You know just from a time perspective, that changes almost everything traditional marketers know about testing.
Why is that so important?
Because, it forces the rest of the company to queue and request to be part of the test, and it has to make a business case.
I'll give you an example. If somebody is in a queue and says, "I want to test to get people at a breakfast event in Brazil and I hope to fill up a room with 20 people that can listen to SAP for breakfast," and somebody else comes and says, "I want to test my new shopping cart for the Web analytics — the shrink-wrap software we sell over e-commerce — and I'm hoping this can generate an incremental $2 million in revenue."
You can immediately see where I'm going with our queuing process. We're going to put testing resources behind revenue generation e-commerce as opposed to the breakfast event that hopes to get 20 people to attend a physical event.
The number one takeaway is very simple.
As hopeful and aspirational as we were, or idealistic as we were when we bought into the power of what we were setting up Ö OK, so we had some aspirations.
We said, we're going to take a leap of faith. We put the platform in place. Then, we went further and put people in place, and I personally went on the line to hire people in lieu of search marketers, email marketers, etc.
I hired test lab people.
We went out and put a new head count in place, and then put the whole process in place for marketing. So, we're out on a limb because we completely believed in what was possible, and weíre now on year three and we still donít think we've scratched the full surface on how much value there is in this process.