The Case for Betting on Long Shots

Nearly 40% of Alzheimer's drug projects over a decade targeted one hypothesis: that beta-amyloid protein was the culprit of disease. Early research looked extremely promising, and pharmaceutical companies began running in the same direction. But billions of dollars later, drugs based on this theory have shown only modest benefits. 

According to a new study from Duke University’s Fuqua School of Business, what looks like a run of bad luck is actually the foreseeable result of what the researchers call “market herding.”

Professor Ashish Arora, along with Professor Sharique Hasan and Ph.D. candidate William D. Miles, found that when markets are dominated by companies that only conduct one experiment at a time, they tend to crowd around the same ‘safe’ approach. This leads to less diversity in the ideas being tested, and a higher chance that everyone fails together. 

In the paper, If You Had One Shot: Scale and Herding in Innovation Experiments, the researchers found that when firms can only run one experiment, this leads to insufficient diversity at the market level, reducing the chances of solving major scientific problems. 

“You would think that more independent solvers would bring more diverse perspectives and will explore a wider range of approaches,” Arora said. “But that’s not always true.”

One big approach, many small tests

The researchers distinguish between approach—the underlying theory of what causes a problem—and implementation—a specific experiment based on that theory. If the approach itself is wrong, every implementation based on it will fail, no matter how well-designed. In the Alzheimer’s drug example, betting on beta-amyloid wasn't one company's mistake—it was a systemic vulnerability, the researchers hypothesized.

If a firm had to choose one experiment, it would rationally pick the safest approach, the one that the scientific community deems the most promising. 

“All these independent, one-shot experimenters tend to herd around the approach that everyone else thinks is most likely to work,” Arora said.

This creates a risk for investors: a portfolio of ten startups all pursuing the same scientific approach has correlated downside. If the approach fails, they all fail together.

The researchers found this pattern in the data. In one example looking at Alzheimer’s drug development between 2007 and 2008, they compared Pfizer’s project portfolio with the projects of a control group of 18 smaller firms. The smaller firms each launched only one project in that period, with 44% of those exclusively targeting beta-amyloid. Meanwhile, Pfizer launched six projects, spreading them across different approaches; only 33% of its portfolio was invested in beta-amyloid. The bigger player diversified, while the smaller players herded.

Why large-scale experimenters diversify

If a firm can run multiple experiments at the same time, it has a reason to spread its bets. Running two experiments based on different theories hedges against being wrong in the same way twice.

“When you have multiple shots, you start worrying about correlated failure,” Arora said. “Diversifying approaches becomes a way to protect yourself.”

The researchers also noted that multi-experiment firms are less likely to pick the single most promising approach every time, which lowers the success rate of any one experiment. “But the point is the market outcome,” Arora said. At the market level, that broader search increases the odds that at least one experiment succeeds.

“It’s not about maximizing the hit rate,” Arora said. “It’s about increasing the chance of solving a scientific problem.”

For companies in acquisition, startups pursuing the same approach you already believe in may add scale, but buying one that thinks differently can add resilience.

A division of labor between startups and large firms

The researchers note that these findings don’t unequivocally point to an argument for bigger firms and bigger labs. Instead, the data show that even small, single-experiment firms play a crucial role in innovation.

One-shot firms are more likely to be the first to test entirely new targets—ideas that haven’t yet entered the mainstream. Large, multi-experiment firms, by contrast, tend to add diversity by more thoroughly exploring known but under-studied approaches.

In other words, innovation ecosystems work best when they combine both types. Startups boost novelty; large firms reduce the risk that promising but riskier ideas are ignored. Neither group alone is sufficient. “It’s a complementarity,” Arora said, “not a competition.”

“When the problems are hard and the uncertainty is real, the goal isn’t to be right more often. It’s to make sure we’re not all wrong in the same way,” Arora said.