A Smarter, Cheaper Way to Choose the Right AI Tool

When a marketing agency uses AI to create an ad, it must decide which large language model (LLM) is the best for the job. Sometimes, choosing the most capable model is overkill: for simple tasks, a cheaper AI may produce an equally useful result. But how can a business find the right balance between quality and cost?

This is a problem AI companies often try to solve with “routing.” When chatbots receive simple queries such as ‘What’s the weather today?’ they can route the request to cheaper, faster models, reducing token cost. A token is the primary unit of text that an AI model processes. It's also the baseline unit that AI services use for billing.

“But given the complexity of those LLM models, a firm may not know in advance which model is the most cost-efficient for the specific request,” said Professor Yehua Wei of Duke University’s Fuqua School of Business.

For such cases, a new working paper by Wei, Fuqua PhD candidate Yan Chen and Alexandre Belloni of Chicago Booth proposes a learning-based “cascading” system.

For each task, the system queries AI models one at a time and evaluates each response before deciding whether to stop or try another model. Over time, it learns which models to query first and when an answer is good enough. Because the system often stops before reaching the most expensive models, it can reduce costs while maintaining output quality.

“A cascading system allows the firm to query a very inexpensive LLM first, even if it is unreliable,” Wei said. “Suppose that model produces a good response only 10 percent of the time. You can still avoid using a more expensive model those times, and those savings can add up.”

Using both theoretical analysis and simulations, they found that their approach achieved a better trade-off between cost and quality than standard alternatives.

Which ad should I pick?

Consider a marketing campaign for T-shirt ads targeting men ages 20 to 25, Wei said.

The agency wants to use generative AI to draft the visuals for the ad, but many AI models are available, each with its own capabilities and costs. Marketers could simply pick the model they judge to offer the best combination of quality and cost — the “routing” approach. 

Or they could use the cascading system designed by Wei and co-authors. This system would start with one model and observe the ad the AI produces — perhaps a picture of a T-shirt front and center. Then it would test a second model and get a second ad — maybe this time a street scene showing people wearing the brand’s T-shirts. The system keeps going until it determines another test is not worth the cost, choosing among the ads it has generated. 

“We need to look at more than one because this is a ‘noisy’ problem. We don’t know the thousands of variations a T-shirt ad concept may have, and we don’t know how well each of the AI tools will perform on this task,” Wei said.

Importantly, he said, the system does not necessarily pick the last model it tested, or the ad produced by the most expensive model. “It should be the ad that makes the most sense, among all the ads from the models,” he said.

“The system is ‘cascading’ through a sequence of models, and it stops when it believes it has enough options,” Wei said.

Learning when to stop

The cascading system is inspired by the classic “Pandora’s Box” problem. You have several boxes, and you don’t know which contains the best prize. Opening each is costly, so the challenge is to decide which box to open next — and when to stop searching and keep the best prize found so far.

However, what is unique to the rapid development of AI is that businesses do not have a good understanding of the performance of each LLM for different tasks, Wei said.

To solve this, the researchers designed a framework that learns through experience with previous tasks. As the system handles a T-shirt ad for a specific audience, it records information about the task, outputs from the different AI models, and the customer response — for example, for an online ad, the click-through rate.

The system then uses similarities between requests to improve its decisions. 

“Suppose the next task is another T-shirt ad aimed at a similar audience, but in a different location,” Wei said. “The system can draw on what it learned from previous campaigns to make a better decision.”

For each new task, the system uses this experience to decide which model to try first and whether to continue. It stops when none of the remaining models appears likely to improve the best result enough to justify its additional cost.

An ‘optimistic’ system 

The researchers’ intuition is to design a system that thinks like an “optimistic” decision-maker.

To learn which models are worth trying, the system must occasionally test models whose value is still uncertain. If it always selected whichever model currently looked best, it might settle too quickly on familiar options and never discover that another model performs better for certain tasks.

And in the process, each attempt provides more information, allowing the system to make better decisions in the future.

“The optimism of our model is crucial for it to learn,” Wei said. “If it never tries an uncertain option, it can never discover how valuable that option might be.”

The researchers tested their approach in 30,000 simulations and compared it with two common alternatives: one that sends requests through a fixed sequence of AIs, from the cheapest to the most expensive, and another that picks one considered the most promising for the task and then sticks with it.

The learning-based cascading system performed better across a wider range of settings. The one-model approach performed poorly in many simulations, while the fixed sequence worked well only under certain conditions. The researchers’ system had its greatest advantage when it was unclear whether trying a cheaper model first would be worthwhile.

The results suggest that the system’s flexibility and its ability to learn from experience can bring its performance progressively closer to that of an ideal, omniscient decision-maker.

A system that doesn’t require training a new AI model

Cascading is most useful when a business repeatedly uses AI for related tasks and can evaluate the results, Wei said. Over time, the system can learn when a less expensive model is likely to be good enough and when it makes sense to go directly to a more capable model.

The approach will not be suitable for every application, he said. Consulting several models can take longer than choosing one immediately. In applications where every millisecond matters, the potential savings may not justify that delay.

“If speed is critical, cascading may be too slow because the system may need to go through multiple models,” Wei said.

But many business tasks — such as creating marketing content, drafting reports or analyzing documents — can tolerate some additional time and reduce costs by avoiding an expensive model when a cheaper one has already produced a strong result.

Adopting the approach would not require a company to develop or train a new AI model, Wei added. Instead, the cascading system works with models the company can already access.

“We are not building another AI model,” he said. “We are developing a smarter way for a company to choose among the models available — and to know when to stop.”

“The savings come not from making individual AI models cheaper, but from using the expensive ones more selectively.”