Plenty of solo founders and planners flip an AI tool's deep-thinking switch on or off, and stretch or shrink its answer length, purely by gut feel. You might let it answer quickly when drafting a quote, then let it think as long as it wants when stress-testing the logic of a business plan. But until now, there's been no good way to check whether that choice actually changes the quality of what comes out.
The common intuition is simple: give the model a generous budget, and it thinks longer and harder, so the answer gets better. That's why many people crank the token budget to the max for high-stakes tasks, while locking in which model to use based on a benchmark leaderboard.
But a recently published paper puts a crack in that intuition. Its central claim: how many tokens you allow a model to reason with can flip which model comes out on top. If you've assumed that simply giving your AI model a generous reasoning budget is enough, this is worth a closer look.
What the Study Did
Researchers Rodrigo Guedes de Souza and Alison R. Panisson set out to test an assumption the industry rarely questions: that a model's ranking should hold steady no matter how its reasoning conditions change. If that assumption were true, a model that tops one benchmark should stay on top under any conditions.
To test this, the researchers varied the token generation budget — the maximum number of tokens a model is allowed to use to produce an answer — across seven levels, from 64 to 4,096 tokens. They applied these conditions to four models and three reasoning benchmarks, running 56,476 inference calls in total. In effect, they compared how well the same models solved the same problems under tight budgets versus generous ones.
What They Found
First, 3 to 19 percent of all questions showed non-monotonic behavior — meaning accuracy actually dropped as the budget increased, in a substantial number of cases. The researchers confirmed this wasn't simply an artifact of truncated answers. Which questions triggered this effect also varied by model: overlap between different models was only 6 to 14 percent.
Second, changing the budget flipped the model rankings across all three benchmarks, and the differences were statistically significant. In other words, the model that performs best when forced to answer briefly can lose out to a different model once it's given plenty of room to think.
Third, in an oracle analysis — assuming you always pick whichever model gets each question right — the complementary effect between models was strongest under tight budgets, opening a gap of up to 27.8 percentage points. Fourth, the researchers built a router that assigns each question to a model based on the budget condition, and it closed 14.1 percent of that oracle gap. Budget-aware information helped within the same domain, adding 1.6 to 5.7 percentage points, but it actually hurt performance by 1.2 percentage points when applied across domains.
What This Means for Your Work
Translate these findings into the daily work of a solo founder or planner, and a few habits are worth reconsidering.
First, it's worth questioning the habit of using the same setup for every task just because a model has a good reputation. The model that produces the best result for a quick draft, where you want a short, fast answer, may not be the same one that does best on a deep review, where you give it plenty of time to think.
It's also worth remembering that cranking up the budget isn't automatically the safe choice. If you routinely max out the token limit for your most important documents, keep in mind that accuracy can actually decline on certain tasks — and review the output accordingly.
Finally, it can pay off to run your own small-scale comparisons of which model-and-budget combination actually works best for each type of task. The complementary effect the researchers found between models suggests that switching tools to match the task, rather than leaning on a single model for everything, can produce a real, practical gain.
A Few Caveats
This study was conducted within a limited scope: four models and three reasoning benchmarks. There's no basis yet to assume that the non-monotonic behavior or ranking reversals observed here would show up the same way with other models, or in other kinds of work — like actual document drafting or customer support.
The router's improvement also varied by condition — it helped within the same domain but actually hurt performance when applied across domains. So optimizing based on budget conditions is not a universal fix that applies cleanly to every situation. The researchers themselves stop short of anything more than suggesting that evaluation methods should factor in budget conditions; they don't declare any particular model or budget setting the right answer.
