SUMMARY: Sometimes how-to articles with many sources are so extensive, the article is split into two parts. That rarely happens with a case study. But, this very deep dive into SAP's online testing and optimization program is one of those unique exceptions.
Last week, we began sharing SAP's Test Lab with you, and this week, we conclude that process with details of actual tests run by the program, and a look at challenges in testing and optimization. Plus, you'll read about the big picture learnings from this impressive effort.
The second we slow down internally, you know people will move away from a testing mindset.
It's tough. It's difficult. It's a lot of change management.
It actually forces marketers to do things differently in one major way, which instead of bringing one idea to the surface, they have to bring two.
And, in many cases, especially with creative concepting, that's twice as much work.
You can't do twice as much work for everything. So, then we start having to talk about the volume of activities we do, or the volume of content we do, maybe has to be turned down a bit so that the quality and the testing can be turned up a bit.
All of that is a really big part of change management, and it never stops.
You get the platform in place, you get the people in place, you get a process.
Now, you really think you're good to go. Right? So, your three-legged stool is locked — people, process, platform. I'm ready to go. And, then what happens? You hit the next blocking point, which is sheer resources to execute the test.
What does it mean? We work in a world of finite resources, and we've got design resources, we've got copy resources, we've got development resources that are in place.
You can imagine one Web experience at a time, whether that's emails, or paid search landing pages, or microsites or core web property websites.
We've got people and teams in place that build us out one Web experience at a time. And, all of the sudden, you come up and go, "I want to do some dramatic testing and I want you to change the layout on everything. There's going to be a full layout, new template." And, the development team comes [back] and says the amount of resources required to do that are the same as rebuilding the corporate website.
Marketing and sales alignment is paramount because for us, our center of gravity is sales results, full stop.
Our center of gravity as corporate marketers at SAP is hitting the revenue target and enabling us to do that as efficiently and as quickly as possible. Right?
Revenue is a quarterly gain, which is why the Test Lab has also been set up on a quarterly cycle, because every quarter, we're trying to generate more leads into the sales team, to [get them to] sell more software.
That is so important because when you come into the queue and you raise the hand and go, "I want to test 'X,'" the barometer to which you're measured against, or the acid test, is impact on revenue.
That's always the center of gravity for the Test Lab when we're making the call to test "this" versus "that" and to choose submissions
Testing is an iterative process, like with the Chinese example with registration forms on mobile devices.
You know testing is iterative but you want [the team] focused on locking down what we call definitive insights that can be shared in the quarterly insights book for the rest of marketing as, "This is a function of digital that allows you to convert at a much higher level."
Our customers prefer it. We've proven it statistically and it's been deployed to all of the areas that we previously tested. To your marketing teams, that's a definitive insight. So, as you move forward with new marketing initiatives, please use these insights and do marketing based on them, because then we're all standing on the shoulders of giants.
You probably could test registration forms forever, right? Drop a field. Add a field. Drop another field. Add another field, on and on, and on it goes.
At some point, it's enough. We've probably got enough insights and enough value from that test we're not going to drive a bunch more. Move onto the next definitive insight, and that's why they're KPI'ed on that.
The number one takeaway is very simple. As hopeful and aspirational as we were, or idealistic as we were when we bought into the power of what we were setting up, we had some aspirations.
We said, "We're going to take a leap of faith." We put the platform in place. Then, we went further and put people in place, and I personally went on the line to hire people — in lieu of search marketers, email marketers, etc. — I hired Test Lab people.
We went out and put new head count in place, and then put the whole process in place for marketing.
So, we're out on a limb because we completely believed in what was possible, and we're now on year three and we still don't think we've scratched the full surface on how much value there is in this process.
There are very few things we do year-on-year that keeps driving exponentially more value, and Test Lab is one of them.
That first year the value just comes gushing out, because we've tested so little previously.
But, it never stops because the world is changing. There are new devices, new customers, new solutions, new products [and] new creative techniques. The world is ever changing and the value just keeps coming.
We're all working with finite resources, we're all working with finite budgets.
So, the corporate culture is very interesting because we're all working with finite budgets, but yet, we're in an environment where we go, "Guess what guys, cost savings is not a dirty word, because if that forces us, as a corporation, to do smarter things, or to do better things, or to do more relevant things, or to just pare down the list to the quality versus the quantity, that's not a bad thing."
That's the corporate culture thatís going on right now at SAP.