Can AI Agents Pick the Better Path? A New Benchmark Says They’re Barely Better Than Guessing

A new benchmark argues that today’s AI agents are weak at a specific, consequential skill: choosing the better direction when a long-running task reaches a fork. The benchmark, Taste-Bench, reports that its best-evaluated frontier model answered 59.7% of those binary decisions correctly, according to the authors’ paper and repository. The Decoder identifies that top system as GPT-5.6 Sol.

This isn’t a claim that agents can’t finish long tasks. It’s narrower. Taste-Bench tests judgment at points where a project could go more than one plausible way, then uses the eventual outcome to label which direction turned out better. The authors call this capability “taste”: recognizing a promising path before its consequences are clear, such as preferring one implementation, experiment, or research direction when both look defensible at the start. The benchmark has 502 decision-fork questions drawn from engineering and research settings.

The number is modest. On a two-option question, guessing gets about 50%. The top model scored 59.7%.

More thinking didn’t help. The authors also report that giving models a larger reasoning budget didn’t improve accuracy. That cuts against a common assumption that an agent facing an ambiguous strategic choice will decide better if it’s allowed to think longer. The material doesn’t explain why. It doesn’t show whether models lack the relevant context, misread signals in the prompt, latch onto surface patterns, or simply can’t anticipate the later consequences that define the labels. And it doesn’t show that extra reasoning is useless in general. The result applies to the evaluated models, these 502 questions, and the researchers’ reasoning-budget settings.

Why this kind of test matters. Ordinary task-completion benchmarks can miss this failure mode. An agent can write competent code and coherent notes while repeatedly making poor high-level choices about what to pursue next. For agents meant to run for hours or days, the job includes choosing approaches, dropping unpromising work, and noticing when a reasonable-looking plan is heading somewhere worse.

What the evidence doesn’t establish. The material doesn’t compare Taste-Bench directly with established agent benchmarks, or show how well a Taste-Bench score predicts production performance. It doesn’t give the error distribution across engineering and research questions, so a single score can’t say whether models fail on subtle calls, highly technical ones, or decisions humans would also find hard. It also doesn’t say how human experts would score on the same questions, which matters for judging whether 59.7% is a weak result or a hard test.

Bottom line: The evaluated frontier systems were only modestly better than a coin flip at picking the historically better branch, and more reasoning budget didn’t change that. That’s a credible caution against giving agents unchecked authority over strategic choices just because they sound fluent or can sustain a long sequence of actions. It isn’t evidence that agents can’t plan, code, or do research. Taste-Bench is most useful as a pressure test: the question for long-running agents isn’t only whether they can keep working, but whether they can repeatedly choose work worth continuing.