EU Think Tank
  • Home
  • Business
  • Leadership
  • Economics
  • Recruitment
  • Innovation
  • Strategy
  • More
    • Customer Experience
    • Managing People
    • Managing Yourself
    • Communication
    • Marketing
    • Organizational Culture
    • Technology
Featured Posts
    • Marketing
    How Marketing Leaders Are Driving Growth in the New Era of Event Planning – SPONSOR CONTENT FROM CVENT
    • June 30, 2022
    • Managing People
    Developing ESG Strategy Is Hard, But Executing It Is Even Harder – SPONSOR CONTENT FROM DAGGERWING GROUP
    • June 30, 2022
    • Organizational Culture
    Are Your Organization’s DEI Efforts Superficial or Structural?
    • June 29, 2022
    • Strategy
    Does Your Strategy Have a Spine?
    • June 29, 2022
    • Managing People
    Supporting Your Team’s Mental Health After a Violent News Event
    • June 29, 2022
Featured Categories
Business
View Posts
Communication
View Posts
Customer Experience
View Posts
Economics
View Posts
Green
View Posts
Health
View Posts
Hiring and Recruitment
View Posts
Innovation
View Posts
Leadership
View Posts
Managing People
View Posts
Managing Yourself
View Posts
Marketing
View Posts
Middle East
View Posts
News
View Posts
Organizational Culture
View Posts
Russia
View Posts
Saudi Arabia
View Posts
Strategy
View Posts
Technology
View Posts
Ukraine
View Posts
Uncategorized
View Posts
EU Think Tank EU Think Tank
7K
9K
4K
1K
EU Think Tank EU Think Tank
  • Home
  • Business
  • Leadership
  • Economics
  • Recruitment
  • Innovation
  • Strategy
  • More
    • Customer Experience
    • Managing People
    • Managing Yourself
    • Communication
    • Marketing
    • Organizational Culture
    • Technology
  • Innovation

“The Power of These Techniques Is Only Getting Stronger”

  • December 9, 2021
  • euthinktank
Total
0
Shares
0
0
0

Jeremy King has worked in technology for nearly three decades—and has spent much of the past 15 years helping companies use experimentation and data to improve decision-making. Now the senior vice president for technology at Pinterest, King spoke with HBR about the benefits of the experimental approach and the kind of culture that’s necessary to support it. Here are edited excerpts of that conversation:

HBR: When did A/B testing first become part of your work?

King: I worked at eBay from 2001 to 2008, and during the second half of my tenure there experimentation platforms and A/B testing became a focal point. In the early days a lot of experiments focused on the company’s search engine. When I worked at eBay, there were 100 million items for sale on the site, and we were constantly trying to optimize what users were shown when they searched on various terms. The goal was to keep it relevant but also to create serendipity. On platforms such as eBay, Etsy, and Pinterest, you don’t want the search function to be too precise. You want to encourage exploration so that people will roam around and discover new things. So we would experiment with different search results and measure things like transactions, click rates, and exploration time to try to get the best mix.

Does a company require a certain kind of culture to excel at experimentation?

To succeed at it, people have to commit to making decisions based on data. For most established companies, that requires transformational change. In many organizations the senior person in the room, the subject matter expert, or the person with direct responsibility makes the decision unilaterally, often based on instinct. At companies that are data-driven, you are much less likely to hear someone say, “My guess is…” or “I bet that….”

When I joined Pinterest, what impressed me most was that 65% of employees there had done a query in its big-data system in the previous six days. They included not just product engineers and executives but people in HR and on the legal team. In our meetings, if somebody asks a question, instead of guessing at an answer, people’s typical response is to flip open their laptops and begin looking through customer transactions to try to find a data-driven answer.

How hard is it for older, nondigital organizations to shift to that kind of environment?

The number one issue at those firms is that people don’t have access to the data. Organizations like to talk about data democracy, but there are barriers, such as privacy concerns. I get lots of questions from people in all kinds of industries who are skeptical: “Should I really allow the entire company to see all this data?” Data democracy requires an investment and a cultural shift, but the benefits you get from letting more of your company have access to your data are significant, because it unlocks better decision-making.

You spent nearly eight years at Walmart. Describe its experimentation culture.

At Walmart, people still talk about Store Number Eight, which was the location Sam Walton used when he wanted to experiment with some new approach. The practice of selecting a small number of locations where you try out new ideas continues: Walmart has approximately 10 stores it designates for experiments, with at least one in each region. Experiments typically involve things like floor layout or interactive devices. When I was at the company we experimented with a store that had only self-checkout aisles and no cashiers. As you can imagine, in a physical store experimentation is much slower than it is in a digital environment. Walmart’s culture is also affected by its merchants, who have so much experience that they sometimes rely more on instinct than on data. That instinctual approach can be successful up to a point. But especially when you’re operating at scale and launching thousands of new products each day, as we did at Walmart.com, one person’s ability to understand every new item coming into a category and at what volume it’s going to sell in each region is limited. That task is better left to a computer.

Do you have to hire different kinds of people to support an experimentation-driven culture?

I’m not sure we hire differently, but it does require a different kind of onboarding. Companies like Facebook, Google, and Pinterest are famous for their long onboarding processes. I have friends at Facebook, where new employees spend two full weeks going through data training so that everyone understands what kind of data is available, how to access it, and how to best use it to support decision-making. That kind of training requires a huge investment.

Can companies become too focused on experiments, to the point that it slows decision-making?

At Pinterest we have that debate internally quite a bit. To avoid letting experiments get in the way of a decision, we use a technique called “holdouts.” Let’s say your gut instinct is that a certain change to the site is going to have positive effects, and you feel very confident of that. Instead of waiting two weeks to do an A/B test, sometimes we’ll make the change right now for 99% of users but not make the change for 1%. Then we test to make sure the experience for the 99% really is exceeding the experience of the small group we’ve held out. That allows us to make a change immediately but still test to be sure our instincts are right.

Do you worry about whether the costs of all this experimentation are worth the investment?

Experimentation is relatively expensive, but the gains we’re seeing are so significant that it’s usually worth it. At Pinterest, for instance, we’ve been running experiments looking at how well machine learning can detect content that goes against our community guidelines. We’ve built a new system that shows a 20% increase in the ability to detect that content. There is no way we’d have been able to get that kind of improvement without testing one approach against another, dozens and dozens of times. For example, we’ve developed new machine-learning technologies to identify and hide self-injury content, so there’s much less of this on Pinterest, and reports of this content from Pinners have decreased by 88% over the last year. If someone does report a Pin for self-harm, we’re now three times faster at removing it, which means fewer people see it. Experimentation helped us achieve that.

Is there a danger that A/B testing is just another fad?

I don’t think so. I began using this approach in 2004, so it’s been 15 years already. The power of these techniques is only getting stronger. The platforms used to run these experiments are becoming more widely available and more efficient. I expect that over time we’ll see more business decisions at more companies being made by doing experiments.

A version of this article appeared in the March–April 2020 issue of Harvard Business Review.

Total
0
Shares
Share 0
Tweet 0
Pin it 0
You May Also Like
Read More
  • Innovation

Four Ways Digital Leaders are Accelerating Their Innovation Strategy – SPONSOR CONTENT FROM EY-PARTHENON

  • euthinktank
  • June 17, 2022
Read More
  • Innovation

Using Fiction to Find Your Strategy

  • euthinktank
  • June 14, 2022
Read More
  • Health
  • Innovation
  • News
  • Technology

The Global Mission to Tackle Cancer

  • Sam Tilston
  • June 14, 2022
Read More
  • Innovation

Identifying Unmet Needs in a Digital Age

  • euthinktank
  • June 14, 2022
Read More
  • Innovation

Think You Have a Great Idea? Ask These 6 Questions to Gain Perspective.

  • euthinktank
  • May 19, 2022
Read More
  • Innovation

What a DAO Can — and Can’t — Do

  • euthinktank
  • May 11, 2022
Read More
  • Innovation

Accenture CEO Julie Sweet on the Most Important Skill Job Seekers Need Today

  • euthinktank
  • May 7, 2022
Read More
  • Innovation

Video Quick Take: Syneos Health’s Larry Pickett on Using Data and Technology to Improve Patient Outcomes – SPONSOR CONTENT FROM PERSISTENT SYSTEMS

  • euthinktank
  • April 30, 2022

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Featured Posts
  • 1
    How Marketing Leaders Are Driving Growth in the New Era of Event Planning – SPONSOR CONTENT FROM CVENT
    • June 30, 2022
  • 2
    Developing ESG Strategy Is Hard, But Executing It Is Even Harder – SPONSOR CONTENT FROM DAGGERWING GROUP
    • June 30, 2022
  • 3
    Are Your Organization’s DEI Efforts Superficial or Structural?
    • June 29, 2022
  • 4
    Does Your Strategy Have a Spine?
    • June 29, 2022
  • 5
    Supporting Your Team’s Mental Health After a Violent News Event
    • June 29, 2022
Recent Posts
  • Are Former Startup Founders Less Hireable?
    • June 28, 2022
  • Help Your Team (Actually) Work Smarter, Not Harder
    • June 28, 2022
  • Inclusive Workplaces Start with Inclusive Leaders – SPONSOR CONTENT FROM BETTERUP
    • June 28, 2022

Sign Up for Our Newsletters

Subscribe now to our newsletter

EU Think Tank
  • Home
  • Privacy Policy
  • Guest Post
  • Contact

Input your search keywords and press Enter.