Editor’s Note: George Sadler’s official title at eBay is global leader for CRM and Loyalty Analytics. But like many titles in marketing these days, even people as smart as George would have a hard time ascertaining what he actually does based solely on what his business card says. Basically, George and his team man a new group that drives significant incremental revenue by testing the hell out of emails, notifications and loyalty messages to know exactly what inspires consumers to participate in auctions. They don’t estimate. They don’t guess. They know.
And because they know, they have re-thought what it means to be a marketer and create marketing-driven growth. This is the second of two “Connectors” posts from George – please call him by his first name, FYI – on the innovative system they’ve developed. Part 1 highlighted the religious adherence to A/B testing that grounds their data-driven decisions, and here he dives into a sales-like compensation model that has altered the eBay culture – for the better. – Brad Young, Global Content Marketing Strategy Leader, Dun & Bradstreet
Two themes at eBay brought us to where our team is today.
First, we are of course an e-commerce platform. We were born and raised as a website company with web design and the scientific method at our core. Here, A/B and other causal statistical tests are common ways of implementing ideas.
Second, analytics teams at eBay have historically reported into the CFO’s organization. We are the first group of our kind to sit outside the CFO function. In this construct, a manager like me would also be responsible for financial planning and analysis. And as you might expect, financial leadership wants to hold teams accountable for the budget they are given.
That perfect storm – a legacy of testing blended with our origins in finance – has worked well for us. It has helped us develop great focus on what genuinely inspires revenue-generating buyer and seller actions. And it has created a dynamic of personal accountability that is very much like what you would see in a sales team.
A Purchase Order For Marketing
In my previous post, I talked about how we don’t just want to know if an eBay user is likely to buy shoes. We want to know if that user will definitely buy shoes only if she receives a certain email message. We determine that with strict A/B testing. When a test works and we put that purchase-driving message into production, the team responsible for the message gets direct credit for “banking” the revenue associated with that successful test. They retire quota.
The comparison I often make is to the purchase order in sales. In sales, you can usually tell with some certainty that if Jane Salesperson had not been there, the company would not have gotten a particular sale. And you know exactly how much that deal is worth, because at the end there is a purchase order. Jane gets credit – retiring quota – for that sale. In our world, we get as close as possible to that same value calculation using statistics and A/B testing. And just like with a sales plan, our marketing teams have revenue quotas for the year that are retired by successful tests that are scaled to the marketplace.
The email team, for instance, does a bottoms-up analysis of what we think we can deliver, which gets married to the company growth targets set by our top management, and then becomes our quota. Then it’s on the team to set quarterly goals needed to reach that quota, and we hold ourselves accountable to it with weekly attainment reporting to P&L and other business owners.
When you unpack that down to the tactical level, we’ll test an idea for a few weeks to measure its associated lift. We test it for however long it takes us to get the signal we need to be confident that the test will deliver results when we scale it into production. Then we put on our finance hats, annuitize the observed test results over a 52-week period, and forecast what incremental revenue the tactic will contribute over the course of a year. The product team puts the tactic into production, and we credit the marketing team that did the test with the projected 52-week incremental revenue. A win email goes out, and the individuals on the team retire the associated quota against their own personal goals, which makes up a significant chunk of their total compensation.
Optimizing this measurement and incentive system is a constant focus. If you are a big believer in testing, you should relish in your failures as much as you do your successes because of the learning opportunities you get from falling short (and fast). That is true for us for sure. One thing we have learned this year is that we realized we needed to decouple our launch testing from quota attainment tests to increase learning velocity. We are doing that in 2015, and here’s why.
When you use the same measurement methodology to learn and improve as you use to determine how much credit you give the team, you do two things. First, you tend to be very conservative when you project the 52-week impact of a test, because you do not want to overstate the incremental revenue impact you expect to have. And second, you delay launching good ideas. Teams will wait to launch a new test until they know it will work and that they will get credit for it. They are incented to run tests longer than they might really need for operational confidence because they want to hurdle the high bar we set for retiring quota.
In 2015, to measure attainment and our contribution to the business, we will use a long-term A/B test over the full year across our entire portfolio of innovations. Individual goal attainments will be measured by this method. Operational launch decisions for individual innovations will be made using shorter A/B tests. By separating these tests, the team will be unconstrained to fail faster and launch more good ideas. When they feel really confident about something, they will pull a test and put it into production perhaps after days instead weeks, because they know, when the financial measures are long-term instead of short-term, they’ll be able to make up for it if they’re wrong. The team is excited because they can regularly look at how the portfolio is delivering and figure out how their volume and velocity can deliver on that.
Basically, you are setting up two currencies. One currency is the long-term financial contribution we are making, and another currency is the results of the tests we are implementing. They are no longer go to be one currency.
Setting this up isn’t easy. The processes involved are likely very different than what you have in place today. But the hardest part is the cultural change that has to come with it.
Managing that starts with helping everyone understand what’s possible. It is an approach that will feel so far outside their current realm that they will struggle to conceptualize how it will work. You have to make it real for not just your team but for your key constituents across the company.
Yes, it creates inter-team competitiveness. Yes, when you have the pod-based structure I described in my last post – where marketers team with statisticians and physicists and economists – you have to account for the different personalities involved. Yet the real magic emerges from that very structure and diversity.
Once you have it established, your team will thrive on it. They will get passionate about the testing process and the stats because the results directly determine the company’s – and their own – bottom line. The team will dig for incrementality and be dedicated to iteration and improvement.
At the end of the day, though, they aren’t salespeople. They are still marketers, and you have to remember that. Guard against the temptation to focus predominantly on pipelines and forecasts. I have been in meetings where we spend most of the time talking about how much revenue we are banking and what the pipeline looks like, and then maybe in the last 10 minutes of the meeting we would address, well, marketing – the creative ideas and executions.
Because no matter how much you evolve your model, it is that piece of the conversation that will always differentiate us from our friends in sales. And we must never lose sight of it.