The 'experimental revolution' in business
From an article by Harvard Business Review
A fundamental shift in the way businesses make decisions is taking place. Traditionally, running experiments and abiding by the scientific method has been the domain of scientists, but managers across industries have been adopting cultures of experimentation to inform their decisions at an unprecedented scale.
At Microsoft’s Bing, for example, approximately 80% of product changes are initially tested as controlled experiments. eBay now saves many millions of dollars per year in advertising thanks to a simple in-house study, which revealed that buying paid search ads wasn’t lucrative for them. Google conducts experiments to see which interview questions are most likely to correspond with on-the-job performance, and then it heavily weights candidates’ answers to those questions.
A combination of the availability of massive amounts of consumer data and improved computing power, plus the fact that it’s easier and cheaper to randomize participants than ever before, has created the perfect conditions for the rise of what’s being called the “experimental revolution” in business. As businesses increasingly demand actionable data to inform their decisions by leveraging experiments, MBA programs need to do more to equip future leaders with the necessary skills.
Instead of calling in managers to solve every puzzle or dispute large and small (Should we make the background yellow or blue? Should we improve basic functionality or add new features? Are staff properly supported and incentivized to provide rapid responses?), teams can run experiments and measure outcomes of interest and, armed with new data, decide for themselves, or at least put forward a proposal grounded in relevant information. The data also provide tangible deliverables to show to stakeholders to demonstrate progress and accountability.
Experiments spur innovation. They can provide proof of concept and a degree of confidence in new ideas before taking bigger risks and scaling up. When done well, with data collected and interpreted objectively, experiments can also provide a corrective for faulty intuition, inaccurate assumptions, or overconfidence. The scientific method (which powers experiments) is the gold standard of tools to combat bias and answer questions objectively.
But as more and more companies are embracing a culture of experimentation, they face a major challenge: talent. Experiments are difficult to do well. Some challenges include special statistical knowledge, clear problem definition, and interpretation of the results. And it’s not enough to have the skillset. Experiments should ideally be done iteratively, building on prior knowledge and working toward deeper understanding of the question at hand. There are also the issues of managers’ preparedness to override their intuition when data disagree with it, and their ability to navigate hierarchy and bureaucracy to implement changes based on the experiments’ outcomes.
The bottom line is that companies today are looking for the person who can do it all — design and run experiments, be numerically literate to interpret the results, utilize their interpersonal savvy to implement data-driven change, and be inspiring and visionary enough to lead a culture of experimentation. These skills, which are at the core of what business students should know when they graduate, demand training in experimentation and the scientific method. But business schools aren’t putting that training at the core of their curricula. What can they do differently?
A course in experimentation should include foundational aspects of the scientific method, which can later be adapted for the specifics of a given firm or industry. The basics include:
-
How to form testable, falsifiable research questions and build on what has been done already
-
How to operationalize constructs and define tasks
-
How to think about random assignment to conditions and threats to the validity of the experiment
-
Why a transparent workflow can allow healthy scrutiny and help protect the experiment’s integrity
-
How to do inferential statistics
Overall, we believe there are three main actions that business schools should take in order to best position students to realize their full potential as leaders of the experimental revolution. First, educate students on how to mine academic literature efficiently and build on insights from prior experimental work, then solidify this as one of many resources they turn to when generating value. Second, teach the scientific method and how to conduct effective experiments in the business world using up-to-date best practices. Finally, give students an opportunity to practice these methods while they’re in school, so they’ll be more likely to apply them once they leave the classroom and enter the workplace.
For centuries, in all sorts of professional arenas, we’ve relied on experimentation and the scientific method to generate reliable and insightful outcomes. But all too often in business today, as we try to cope with the onslaught of change, we rely on costly and wild intuition-based decision making, and it’s costing us.
Read the full article here.
Retweet about this article:
From an article by Harvard Business Review, 01/09/2021