5x the Cost. 1 Win Out of 4. Are AI Agent Teams Worth It?

Vals AI reports that multi-agent coding teams cost 1.8 to 5.1 times as much as solo agents on its Vibe Code Bench, yet only one of four direct team-versus-solo score comparisons showed a statistically significant difference. The result challenges a common assumption: that adding more agents reliably improves coding quality enough to justify the extra model spend.

Vals AI compared solo-agent and team-based setups on the benchmark. The narrow conclusion the evidence supports is that across the configurations it tested, teams didn’t consistently produce a measurable quality advantage over a single-agent baseline, despite costing more. The Decoder describes the finding the same way: multi-agent arrangements substantially raised costs while generally failing to show a statistically meaningful edge.

“Not statistically significant” isn’t the same as “no benefit.” It means the score differences in three of the four comparisons weren’t large enough, given the data, to rule out chance. It doesn’t prove the teams were no better.

It also doesn’t show that multi-agent systems don’t work. The material available doesn’t detail the benchmark’s individual tasks, prompts, agent roles, stopping conditions, pricing assumptions, or confidence intervals for each comparison. It doesn’t show whether teams offered benefits outside the measured score, such as artifacts that make human review easier or broader testing. Team design matters too. Several agents independently proposing an implementation is a different arrangement from one agent writing code while others test and revise it, and one benchmark can’t say which designs, if any, are worth the cost for a given repository or process.

Bottom line: Agent count alone isn’t evidence of better coding performance. With teams costing up to five times as much, the practical question for anyone building or buying agent workflows is which task types and orchestration designs earn back that cost, and for now the benchmark suggests that has to be measured case by case, not assumed.