A swarm of agents, trained on reward: better answers, five times the cost

When an agent struggles with a hard, open-ended task, the popular answer is to use more agents: a lead that plans, specialists that work in parallel, a synthesis at the end. We had an environment where we could check that claim rather than assume it, so we did.

The short version: a team of agents is not automatically better. Untrained, the team scored lowest of the five agents we have now measured in this environment. Trained on the environment’s reward — every member keeping its own playbook of lessons — it became the most accurate: 80% of held-out investigations fully correct against the single trained agent’s 70%, with the best evidence quality of any arm and not a single decoy claimed. It also spent 4.9 times the tokens to get there, which on the scoring rubric cancels the gain exactly.

This continues the pilot published yesterday, and uses the same company, the same 20 held-out investigations, the same tools and the same grader. Only the thing holding the tools changed.

Objective

In the first pilot, a single agent investigated: it chose what to query, read the results, formed hypotheses and submitted a conclusion. Training it on reward made it dramatically more efficient at the same accuracy.

A team is a different bet. Three specialists can examine three explanations at once, in separate contexts, so more evidence reaches the conclusion and no single context has to hold all of it. That is the argument. The counter-argument is just as plausible: three specialists produce three partial stories, and someone has to decide between them without having seen the data themselves.

We wanted to answer two questions with the same measurements we already had:

1. Does dividing an investigation among specialists beat one agent doing all of it?
2. Does learning from reward help a team more or less than it helps a single agent?

Plan

The team is a lead investigator and three specialists. The lead reads the case, splits it into three lines of inquiry and assigns one to each specialist. The specialists work at the same time, each in its own context, with the same tools the single agents had, minus the ability to submit: only the lead concludes. Each reports back what it found, the numbers behind it, and what it ruled out. The lead names what is still missing, can send one follow-up round, and then submits one conclusion, which is what gets graded.

What learns. In the trained arm, each member keeps its own playbook, and there is one shared team playbook that everybody reads. Reward arrives only at the end, for the team’s single conclusion, so credit has to be assigned. We do it the same way the single-agent learner does — contrast a high-reward investigation with a low-reward one on the same case — but show that contrast one member at a time, through only that member’s own turns. Lessons about dividing work and handing over go to the shared playbook instead. The model’s weights never change; the playbooks are the whole checkpoint, and you can read them.

What stayed fixed. The company, the 20 held-out investigations, the four test types (familiar cause types, two causes at once, cause types never trained on, and a decoy planted to mislead), the tools, and the grader — including its efficiency penalty, which charges for every tool call beyond fifteen. A team makes more calls than one agent, so that penalty costs it more. We left it alone: the cost of an answer is part of the answer.

Experiments

Two new arms, both on the same 20 held-out tasks: the team as prompted roles with no learning, and the same team trained on the same 12 training cases and the same budget the single trained agent got.

Before either arm ran, we tried one investigation and read the whole transcript. It caught a flaw of our own making: specialist reports were being cut to a third of their length before the lead read them, and the lead said so in its own review — it complained that the reports it had been given were truncated. Reports now reach the lead whole.

One flaw we found later we deliberately did not fix: see the over-claiming in the outcomes below. Both swarm arms had to run identical prompts, or the comparison between them would mean nothing.

Everything was re-graded together with the three arms from the first pilot, so all five are scored by the identical verifier.

Outcomes

“Accuracy only” is the same rubric with the efficiency penalty removed, to separate how right an arm was from what it spent.

Training the team is the clearest effect we have measured in this environment. Trained against untrained, on the same 20 investigations: +0.163 reward, 95% confidence interval [+0.039, +0.317], p = 0.023, and fully-correct investigations up from 60% to 80%. Unlike every comparison in the first pilot, this one is measured across all 20 tasks rather than 5, which is why it can clear the significance bar at all.

The team did not beat one trained agent. On the rubric they are indistinguishable: 0.798 against 0.820, p = 0.75. The team is genuinely more accurate — it identified causes better (F1 0.97 against 0.87), sized them better (attribution 0.90 against 0.76), and was the only arm that never once claimed a decoy — and it paid for that with 185k tokens per investigation against 38k.

Why the untrained team was the worst arm of all. It had the same tools as arm B and scored well below it. The transcripts show a specific failure: three specialists each report findings from their own slice, and an untrained lead passes them all through instead of deciding between them. It claimed 3.4 causes per investigation against the single trained agent’s 1.4. In the worst case it split one cause into sixteen slices by category and channel, each with a tiny share, together adding to 0.17 of the decline instead of 1 — the grader gave it full marks for finding the cause and nothing for sizing it.

Each member learned something only it could use. The lead learned to separate demand from post-order execution before naming anything; the demand analyst to put booked demand beside completed revenue, which tells “customers stopped buying” apart from “orders stopped completing”; the supply analyst to localise to a single warehouse and require three aligned signals before concluding; the pricing and data analyst to check whether the reporting pipeline itself explains the gap before blaming the business. The shared playbook holds only coordination: assign each analyst a competing explanation rather than a domain, make every handoff decision-ready, do not stop at the first large driver.

What training cost. It fixed six investigations and broke two, and both regressions have the same signature: the trained team adds a partner-loss claim that is not a true cause, because its specialists are now told to report partner and customer activity and the lead sometimes promotes that report into a claim. Learning from reward changes behaviour in more places than the one you were aiming at, which is why the held-out set and the per-task diff matter more than the headline average.

Conclusion

If you are choosing between one agent and a team for this kind of work, the pilot says: a team is worth it only if you train it, and only if accuracy is worth about five times the cost. An untrained team of capable agents was worse than one capable agent — not because the model is weak, but because dividing work creates a decision problem that nobody in the team is responsible for solving. Training put that responsibility somewhere: the lead learned to pick, and claims per investigation fell from 3.4 to 1.65.

The result we did not expect is where the benefit landed. The team’s edge shows up on the two hardest test types — misleading decoys (80% against 60%) and familiar causes it had been getting wrong (100% against 80%) — and vanishes on combined causes, where the plain SQL agent’s 60% is still the best any arm has managed and the four since have all sat at 40%. Two causes at once remains the unsolved case here, for one agent and for four.

Both arms together cost 10.2M tokens and 46 minutes.

Next, in order: tell the lead to consolidate — one claim per cause, at the narrowest scope the evidence supports, shares summing to about 1 — which addresses both the over-claiming and the spurious extra claim; then a cost-matched arm that caps the team at the single agent’s token budget, because “is a team better at equal cost?” is the question these numbers actually pose; then the same comparison with a cheaper model, where a team of inexpensive agents against one expensive agent is the trade that would matter in production.

As with the first pilot, five tasks per test type is too few for any per-type difference to be significant, and this is one seed and one model. The full test — two models, three agents each, 100 held-out tasks — is still planned, and its results will be published whether or not they support what we found here. If you are working on agents for investigation or root-cause analysis, we would like to hear from you.

All notes