Jeremy King has worked in technology for nearly three decades—and has spent much of the past 15 years helping companies use experimentation and data to improve decision-making. Now the senior vice president for technology at Pinterest, King spoke with HBR about the benefits of the experimental approach and the kind of culture that’s necessary to support it. Here are edited excerpts of that conversation:
HBR: When did A/B testing first become part of your work?
King: I worked at eBay from 2001 to 2008, and during the second half of my tenure there experimentation platforms and A/B testing became a focal point. In the early days a lot of experiments focused on the company’s search engine. When I worked at eBay, there were 100 million items for sale on the site, and we were constantly trying to optimize what users were shown when they searched on various terms. The goal was to keep it relevant but also to create serendipity. On platforms such as eBay, Etsy, and Pinterest, you don’t want the search function to be too precise. You want to encourage exploration so that people will roam around and discover new things. So we would experiment with different search results and measure things like transactions, click rates, and exploration time to try to get the best mix.
Does a company require a certain kind of culture to excel at experimentation?
To succeed at it, people have to commit to making decisions based on data. For most established companies, that requires transformational change. In many organizations the senior person in the room, the subject matter expert, or the person with direct responsibility makes the decision unilaterally, often based on instinct. At companies that are data-driven, you are much less likely to hear someone say, “My guess is…” or “I bet that….”
When I joined Pinterest, what impressed me most was that 65% of employees there had done a query in its big-data system in the previous six days. They included not just product engineers and executives but people in HR and on the legal team. In our meetings, if somebody asks a question, instead of guessing at an answer, people’s typical response is to flip open their laptops and begin looking through customer transactions to try to find a data-driven answer.
How hard is it for older, nondigital organizations to shift to that kind of environment?
The number one issue at those firms is that people don’t have access to the data. Organizations like to talk about data democracy, but there are barriers, such as privacy concerns. I get lots of questions from people in all kinds of industries who are skeptical: “Should I really allow the entire company to see all this data?” Data democracy requires an investment and a cultural shift, but the benefits you get from letting more of your company have access to your data are significant, because it unlocks better decision-making.
You spent nearly eight years at Walmart. Describe its experimentation culture.
At Walmart, people still talk about Store Number Eight, which was the location Sam Walton used when he wanted to experiment with some new approach. The practice of selecting a small number of locations where you try out new ideas continues: Walmart has approximately 10 stores it designates for experiments, with at least one in each region. Experiments typically involve things like floor layout or interactive devices. When I was at the company we experimented with a store that had only self-checkout aisles and no cashiers. As you can imagine, in a physical store experimentation is much slower than it is in a digital environment. Walmart’s culture is also affected by its merchants, who have so much experience that they sometimes rely more on instinct than on data. That instinctual approach can be successful up to a point. But especially when you’re operating at scale and launching thousands of new products each day, as we did at Walmart.com, one person’s ability to understand every new item coming into a category and at what volume it’s going to sell in each region is limited. That task is better left to a computer.
Do you have to hire different kinds of people to support an experimentation-driven culture?
I’m not sure we hire differently, but it does require a different kind of onboarding. Companies like Facebook, Google, and Pinterest are famous for their long onboarding processes. I have friends at Facebook, where new employees spend two full weeks going through data training so that everyone understands what kind of data is available, how to access it, and how to best use it to support decision-making. That kind of training requires a huge investment.
Can companies become too focused on experiments, to the point that it slows decision-making?
At Pinterest we have that debate internally quite a bit. To avoid letting experiments get in the way of a decision, we use a technique called “holdouts.” Let’s say your gut instinct is that a certain change to the site is going to have positive effects, and you feel very confident of that. Instead of waiting two weeks to do an A/B test, sometimes we’ll make the change right now for 99% of users but not make the change for 1%. Then we test to make sure the experience for the 99% really is exceeding the experience of the small group we’ve held out. That allows us to make a change immediately but still test to be sure our instincts are right.
Do you worry about whether the costs of all this experimentation are worth the investment?
Experimentation is relatively expensive, but the gains we’re seeing are so significant that it’s usually worth it. At Pinterest, for instance, we’ve been running experiments looking at how well machine learning can detect content that goes against our community guidelines. We’ve built a new system that shows a 20% increase in the ability to detect that content. There is no way we’d have been able to get that kind of improvement without testing one approach against another, dozens and dozens of times. For example, we’ve developed new machine-learning technologies to identify and hide self-injury content, so there’s much less of this on Pinterest, and reports of this content from Pinners have decreased by 88% over the last year. If someone does report a Pin for self-harm, we’re now three times faster at removing it, which means fewer people see it. Experimentation helped us achieve that.
Is there a danger that A/B testing is just another fad?
I don’t think so. I began using this approach in 2004, so it’s been 15 years already. The power of these techniques is only getting stronger. The platforms used to run these experiments are becoming more widely available and more efficient. I expect that over time we’ll see more business decisions at more companies being made by doing experiments.
