Tech culture often sees failure as essential. Facebook founder Mark Zuckerberg famously described the mindset as “move fast and break things.”
But does that mantra of rapid experimentation actually help companies succeed?
To answer that question, Sharique Hasan and Aaron “Ronnie” Chatterji, both strategy professors at Duke University’s Fuqua School of Business, studied more than 35,000 high-tech startups founded between 2008 and 2013. They summarized their findings in “Experimentation and Startup Performance: Evidence from A/B Testing,” which is forthcoming in the journal Management Science and co-authored by Rem Koning of Harvard Business School.
The researchers found startups that use a series of randomized controlled trials to test options for product features – otherwise known as A/B testing – are more successful, on average, than startups that don’t follow this type of data-driven approach, Hasan said in a live discussion of the research on Fuqua’s LinkedIn page.
Going beyond gut instinct
“To be honest, most startups and most small businesses have a pretty standard framework for making decisions, which is ‘trust your gut,’” Hasan said. “Maybe collect a little bit of data. But usually, entrepreneurs have a belief about something that [their] startup should do, and they pursue it and they follow through,” he said.
A/B testing allows for more quantitative decision-making that also hastens the discovery of what works and what doesn’t, Hasan said.
“If an idea is getting traction, you can scale it faster, and if it’s not going anywhere, you can kill it faster,” he said.
For their analysis, the researchers studied the characteristics, web metrics and product innovations of thousands of startups that did and did not use A/B testing to determine whether and how testing helped them scale their products and companies.
Firms that adopted A/B testing improved their performance, as measured by page views, by 30 to 100 percent after a year of use. Those using A/B testing also introduced new products at higher rate (9 to 18 percent) than firms that did not experiment, according to the research.
A strategy that scales
A common notion among firms is that A/B testing is effective only for incremental changes or small-scale questions, Hasan said, such as Google’s test of 41 shades of blue to identify the best color for its search toolbar.
Rather, he said, the qualitative research and data in the paper showed A/B testing could also guide more significant decisions as part of a firm’s overall strategy, and that testing increased the chances those decisions would be successful, Hasan said.
“The scientific method applied to business decision-making can be a very powerful tool,” Hasan said. “A/B testing is just one instantiation of actual data-driven decision-making by firms that can have a powerful effect on the quality of a decision that a startup makes.”
An underutilized tool
What surprised the researchers was that, despite these clear benefits, less than a fifth of firms were using A/B testing. Testing was more frequent in Silicon Valley, where firms used it about a quarter of the time, compared to 19 percent at firms outside the San Francisco Bay Area. Firms with 10 or more employees were more likely than smaller firms to use the strategy.
The data also suggested that earlier-stage, small startups without financing could benefit the most from A/B testing, but unfortunately, Hasan noted, these firms used testing the least – only 13 percent of the time compared to 25 percent for more established firms financed by angel or venture capital investors.
“This kind of supports this idea that decision-making may be an important bottleneck for startup success, and that using a more scientific, rigorous approach to start decision-making can really improve quality of a startup’s performance,” he said.
The earlier firms can implement a way to manage what could be hundreds of potential ideas, the better their chances of success, Hasan said.
“[A/B testing] opens up ideation inside a startup,” Hasan said. “With advances in technology that allow startups to potentially test thousands of different options, A/B testing is both a cost-effective and rapid method for discovering startup product viability.”