Note · 2026-09-22

When a team of agents beats one: the weekly performance review

In our last two notes, teams of agents did not beat one well-trained agent. The reason was the same both times: the job fitted inside one agent. In the investigation arena, a single agent answered in 7 to 10 of its 25 turns, so extra agents had nothing to do but repeat its work. If teams of agents are ever worth their cost, it should show on work that does not fit: many separable pieces, a few causes that cut across them, and a deadline. So we built such a job, stated in advance where a team should win and lose, and trained both.

Trained on the same reviews, with three tool calls per step, the team beat the single agent on all ten held-out reviews when the deadline was tight: 0.509 reward against 0.350 over three runs each (p = 0.002). With time to spare they were level, and untrained the team was no better than one agent. Looking for the reason undid the headline. Training had made the single agent thorough, and at three calls a step thoroughness left a third of the material cells with no driver named; a learner told about the deadline did not change that. Given more tool calls per step, and nothing else, the same trained agent drew level with the team at six and led it at nine (0.577 against 0.509), at a third of the tokens. The team’s advantage was parallel tool capacity, and one agent can have that. What one agent cannot get from more calls is the team’s context. When the evidence outgrew one agent’s window, its reward fell below zero and most of its numbers were invented, while the team, holding its facts on a shared board, lost 0.15.

Objective

The use case is one every finance team runs: the weekly group performance review. Revenue for each business unit in each region is compared with plan, every material miss is explained with evidence, causes that span several cells are identified once, and the whole thing is rolled up for the executive committee by a fixed time. It is wide by construction, and the pieces are mostly separable, which is exactly where the case for a team of agents rests.

We wanted a rule an enterprise can use: for which analytics jobs should you train one agent, and for which a team? That needs an environment where a single agent does an “okay” job and a team has room to do better, a reward that cannot be gamed, and a comparison where both arms are trained.

Plan

The environment. A generated company with four business units (computing, networking and storage, software and services, accessories) across four regions: 16 cells, each with a weekly plan and actuals. The world is generated with its causes injected, so the grader knows the answer exactly: local causes (stock-outs, price increases, competitor launches), causes that span cells (a promotion error across a region, a warehouse outage, a lost partner, a payment failure, a price policy), plan errors that only planning documents reveal, and decoys that look like causes and are not. A cell is material when it misses plan by more than 4%. The agents query it through ten data tools: SQL, cell summaries and breakdowns, the plan history, documents, events and a revenue decomposition. Every result carries an id, and every claim must cite the ids that show it.

Left: a four-by-four grid of business units by regions for one held-out review, with ten material cells outlined in red and labelled with their true cause: a promotion error runs down the whole EMEA-North column, a plan error sits inside the largest miss (Computing, EMEA-North, minus 26%), and competitor launches and stock-outs explain the rest. Right: the two arms on a 12-round time axis. One agent works all 12 rounds alone. The team has a lead for round 1, four analysts and a cross-cell analyst working in parallel for rounds 2 to 10, and a reconciler composing the review in rounds 11 and 12. — Figure 1. One held-out review, and the two arms under the same deadline. The team spends three of its twelve rounds on planning and composing; parallelism has to pay for that.

The reward. The grader parses the structured review and never asks a model for an opinion.

R = Q − P − C

Q = 0.60·Q_cells + 0.20·Q_shared + 0.10·Q_reconcile + 0.10·Q_group
  Q_cells      each material cell: drivers found (F1), amounts right,
               evidence verified; cells weighted half by the size of
               the miss, half equally
  Q_shared     causes spanning cells, listed once with the right cells
  Q_reconcile  drivers plus residual add up to the variance,
               residual under 20%
  Q_group      the largest drivers named in the group roll-up
P = decoys claimed, claims whose cited evidence does not check out,
    invented shared causes, over-attribution, drivers claimed in
    immaterial cells; 0.5 for no review at all
C = 0.10 · tokens / 1M + 0.05 · rounds used / deadline

Two design choices carry the experiment. Half of each cell’s weight is equal, so an agent that explains only the biggest misses scores about 0.66, not 0.9: the long tail counts, as it does in a real review. And a claim earns full credit only if the query result it cites really touched the right table and named the right scope. Before any agent ran, eight degenerate strategies (blame the plan for everything, claim every kind of cause everywhere, plausible guesses with no queries behind them) were run against the grader; the best of them averages 0.03, while the true answer scores 0.997.

The arms. One agent with every tool, against a team of seven: a lead that assigns cells from the summary, four analysts working their cells in parallel, a cross-cell analyst looking for causes that span cells, and a reconciler that composes the one review from what they post. Both use the same frozen model and face the same limits: a deadline in rounds, at most three tool calls per round, and a 128k-token context per agent. Training writes playbooks, not weights: one for the single agent; for the team, one per role and one they all read.

What we expected, written down before building: a clear team win when the review is wide and the deadline tight; a narrower gap when there is time to spare; nothing for a team to add on a single cell.

Experiments

Model	`gpt-5.4`, frozen; learning is playbooks only
Conditions	16 cells with a 12-round deadline (where training happened); 16 cells with 30 rounds; one cell with 30 rounds
Training	12 training reviews, two attempts each for the learner to contrast, two update steps; identical for both arms
Held-out evaluation	10 new reviews at 16 cells and 12 rounds: untrained once, trained three times per arm per review. 5 reviews each for the other two conditions
Controls	untrained: one agent working one cell at a time, and the team’s structure with a single analyst
Follow-ups	the single agent trained again, by a learner told how each attempt spent its budget; and the capacity curve: the trained single agent with 6, 9 and 12 tool calls per round, same deadline, no retraining, with the untrained agent at 9 as a control; and the context limit: every agent’s context budget shrunk to 12k, 8k and 6k tokens
Statistics	paired over reviews: bootstrap intervals and exact sign-flip tests
Cost	81.3M tokens across all phases

Outcomes

Untrained, the team was no better than one agent

With a tight deadline, the untrained team scored 0.360 and the single agent 0.408 on the ten held-out reviews (difference −0.049, p = 0.56). Our first look, on five reviews run once, had the untrained team ahead by 0.145 on all five. It did not replicate, because the same agent on the same review varies a lot from run to run: the team scored 0.575 on one review in one run and 0.038 in the next. That is why every trained comparison below uses three runs per review.

The controls, run untrained on five reviews each, pointed the same way. One agent working one cell at a time, with a fresh context for each, was the worst arm in every condition: fixed budgets per cell waste turns, and it never identified a cause that spans cells. The team’s structure with a single analyst lost to the plain single agent under the deadline: the lead and the reconciler cost three of its twelve rounds, and only parallel analysts pay that back.

Trained, at three calls a step, the team won every review

Dumbbell chart of net reward, untrained to trained, for one agent and the team in three conditions. 16 cells, 12 rounds, where both were trained: one agent 0.408 to 0.350, team 0.360 to 0.509, team minus one agent plus 0.16. 16 cells, 30 rounds: one agent 0.400 to 0.602, team 0.464 to 0.643, plus 0.04. One cell, 30 rounds: one agent 0.822 to 0.904, team 0.896 to 0.962, plus 0.06. — Figure 2. The same playbooks in three conditions. Training moved the team ahead only where the deadline binds; given time, the trained single agent caught up.

After training, the team scored 0.509 against the single agent’s 0.350, a lead of 0.158 (90% interval 0.12 to 0.20). It was ahead on all ten reviews (p = 0.002), and run against run on the same review it scored higher in 82 of 90 pairings. The team’s gains came from what the reward was designed to price: its penalties fell from 0.117 to 0.026, and it found shared causes far more often than the single agent (0.38 against 0.11 on that term). Tokens were not what decided it: the team used 360k per review against 110k, which cost it 0.025 of reward. Tokens would have to cost about seven times more for the single agent to draw level.

Dot plot of every trained run in the 12-round condition: three runs per arm on each of ten held-out reviews, with each review's mean marked. The team's mean is above the single agent's on every review; individual runs of one review vary by up to 0.39. — Figure 3. Every trained run. Single runs are noisy; the team's mean was higher on every review.

Training made one agent thorough, and thoroughness cost it cells

Training did not help the single agent under the deadline (0.408 before, 0.350 after, p = 0.31). Its playbook explains why. It learned a careful routine for every cell: find the narrowest slice that moved, decompose it into price, volume and fulfilment, confirm the cause from an event or a document, then test whether the cause spans cells. Those are good lessons, and they take three or four calls per cell. Twelve rounds of three calls cannot finish eight to fourteen material cells that way, and its coverage of the review fell from 0.79 to 0.67.

Bar chart of coverage, the weighted share of material cells where a true driver was found. With 16 cells and 12 rounds: one agent 0.79 untrained, 0.67 trained; team 0.81 untrained, 0.83 trained; all using about 11 rounds. With 16 cells and 30 rounds: one agent 0.93 in 15 rounds untrained, 0.92 in 20 rounds trained; team 0.97 and 0.98 in about 14 to 15 rounds. — Figure 4. Under the deadline, the trained single agent explained fewer cells. With 30 rounds the same playbook explained them all, and used the extra time.

With 30 rounds, the same playbook was worth +0.20: the agent used 20 rounds instead of 15, kept its coverage, and scored 0.602. At three calls a step, one agent has to trade depth for breadth when time is short. A team does not: its analysts learned the same evidence chain, and also to “use tool budget tightly”; its lead learned to give one analyst the cells that might share a cause; its reconciler learned to tie out every cell and every shared cause exactly.

The lessons carried over

Both arms were trained only at 16 cells and 12 rounds. Their playbooks also improved every other condition we tried them on: with 30 rounds, +0.20 for the single agent and +0.18 for the team; on a single cell, +0.08 and +0.07, where the single agent’s reviews now add up (0.52 to 0.84 on reconciliation) because it learned the arithmetic it used to get wrong. With five reviews per condition these are directions rather than significant results; the team’s improvement with 30 rounds was positive on all five.

A learner told about the deadline did not close the gap

The obvious objection is that the single agent’s learner never knew about the deadline. So we built one that does. For every attempt it sees how the budget was spent: turns and calls used, and what became of each material cell, largest first (explained, wrong, or left empty). It is asked to judge the use of the budget as well as the analysis, and it is not told what strategy to use. Trained on the same reviews in the same way, it learned to triage. Its first lesson is a coverage plan: one localising call for every material cell before any deep dive, and the last turns kept for submitting.

It scored 0.333 against the plain learner’s 0.350 (p = 0.68), and the team beat it on all ten reviews, by 0.176 (p = 0.002). Allocation was never the problem. Every single agent used its whole budget, and nearly every material cell appeared in several of its query results. What training changed is what the agent did with a cell it had looked at:

single agent	a true driver found	drivers claimed, none right	no driver claimed
untrained	70% of material cells	8%	23%
trained, plain learner	55%	10%	34%
trained, learner told about the deadline	57%	8%	35%

Both learners taught the same rule: name a driver only when a tool directly shows the mechanism, and otherwise leave the variance unexplained. It is a good rule, and it costs a confirming call per driver. One agent with 36 calls cannot follow it for eight to fourteen cells; four analysts with 108 calls between them can. The team’s advantage is capacity, not planning.

There was a second lesson. The new playbook says “within the 12-turn budget”. Given 30 rounds, the agent still stopped after about 12 and scored 0.417, where the plain playbook, which knew nothing of deadlines, used 20 rounds and scored 0.602. A lesson about a budget is a lesson about one budget.

Give one agent the team’s capacity, and the team’s lead disappears

If the team’s advantage is capacity, a single agent given more capacity should close the gap. So we took the trained single agent, changed nothing but how many tool calls it may make per round, and ran it again on the same ten reviews, three times each: same deadline of twelve rounds, same playbook, no retraining.

Line chart of net reward against tool calls per round for the trained single agent under the 12-round deadline: 0.350 at 3 calls, 0.488 at 6, 0.577 at 9 and 0.535 at 12, each using about 110k to 120k tokens per review. A dashed reference line marks the trained team at 0.509 with 360k tokens. Open circles mark the untrained single agent: 0.408 at 3 calls and 0.467 at 9. — Figure 5. The same trained agent with more tool calls per round. It draws level with the team at six and passes it at nine, at a third of the tokens.

With six calls a round it drew level with the team (0.488 against 0.509, p = 0.36). With nine it led: 0.577 against 0.509 (p = 0.051, ahead on 8 of 10 reviews). On quality minus penalties the two were level (0.628 against 0.590, p = 0.22); the single agent’s lead on reward is cost, since it used 120k tokens a review against the team’s 360k, and 60 data queries against 87. It found the causes that span cells as often as the team did (0.37 against 0.38 on that term), which at three calls it almost never managed (0.11). Twelve calls did slightly worse than nine (p = 0.02).

The trained playbook was right all along; it was unaffordable. With six calls or more, the share of material cells left with no driver named fell from a third to about a fifth, the team’s level, and penalties stayed low. The untrained agent gained much less from the same calls (+0.06, p = 0.52): it named more drivers, more of them guesses, and its penalties more than doubled.

Past its context, one agent breaks; the team bends

A team’s other possible advantage is context: each member holds only its own share of the evidence, while one agent must hold all of it. Our reviews are small, so rather than generate bigger evidence we shrank every agent’s context budget equally, which is the same as evidence ten to twenty times larger against a 128k window. Over budget, the harness drops the oldest tool results first, for every agent alike.

Line chart of net reward against each agent's context budget. The single agent at 9 calls per round scores 0.577 unlimited, 0.429 at 12k tokens, minus 0.146 at 8k and minus 0.540 at 6k. The team scores 0.509 unlimited, the same at 12k where its context never binds, and 0.358 at 6k. Below, the share of cells whose variance the review states correctly: one agent 100%, 39%, 10% and 2%; the team 100% throughout. — Figure 6. Shrinking every agent's context. The single agent's reward falls below zero and its numbers go wrong; the team loses 0.15 and keeps every variance right.

The single agent, at its best setting of nine calls a round, fell from 0.577 to 0.429 at 12k tokens (p = 0.006), and below zero at 8k and 6k (−0.146 and −0.540). The team at 6k lost 0.151, to 0.358, and beat the single agent on every review at both 8k and 6k (p = 0.002). At 12k, where the team’s context never binds, it led by 0.079 (p = 0.02).

The single agent did not just forget; it invented. Its first result is the portfolio summary, with every cell’s variance, and as the oldest result it is the first to go. It then wrote variances from memory: at 12k only 39% of cells had the right variance, at 6k 2%, and at 6k it claimed drivers worth more than the variance in a third of material cells. The team kept its facts outside any one context. Each analyst holds only its own cells, what it reports goes on the board, and the reconciler composes the review from the board. At 6k the team still stated every variance correctly.

Reading the record is how we found every error

Every run is saved with every query, message and submission, and each of the following was found by reading those records rather than the scores:

A crash that only hurt single agents. A review submitted without its roll-up section raised an error instead of being graded. The team’s reconciler had a fallback; a single agent on its last turn did not, and scored as if it had submitted nothing. It is fixed, and the affected arm was re-run.
The analysts’ playbook would have learned from nothing. The four analysts are named analyst_1 to analyst_4 but share one playbook, and the learner was looking for messages from analyst. We also found that the team’s learner, unlike the single agent’s, was not shown the grade breakdown. Both were fixed before any training.
A single agent could lose a finished review to the clock. It is now reminded two turns before the deadline, as a team’s reconciler always composes at it.
The grader’s early holes, found by the degenerate strategies before any agent ran: pure size weighting let the largest half of cells score 0.90, uncoupled reviews gave free credit for shared causes, and a scope named in a SQL comment passed as evidence. All were closed first.
Found after publication (2026-09-23): documents were read only to their first line. The tools cut every text cell in a result to 77 characters. Every plan memo the agents retrieved was cut before its deciding clause (“…, plus 13% for a promotion”, or a note that nothing was assumed), and 38% of event rows lost their ending. It hit every arm alike, and the agents spent almost none of their calls working around it. Searches now show documents in full, and we re-ran the headline comparison with them: the same ten reviews, the same trained playbooks, the same model version, one run each. At three calls a step the team still leads the single agent, by 0.10 rather than 0.16. With nine calls the single agent still leads the team, by 0.06. Each arm’s reward moved by less than run-to-run noise. What changed most was how often plan errors were found: 70% of them, against 38%, for the single agent at nine calls.

Conclusion

A team of agents beat one agent here for one reason: it could make more tool calls before the deadline. It was not the talking and not the structure: a team with one analyst lost to a plain single agent, and untrained teams were no better than one. And it was not having several minds: given the same parallel capacity, one trained agent matched the team, and at nine calls a step it led, at a third of the tokens. The second reason is context, and more calls do not buy it: when the evidence outgrew one agent’s context, the agent broke, and the team, whose facts sit on a shared board rather than in any one context, only bent.

Training is what made that visible. Trained at three calls a step, the single agent became better at each cell and worse at the review, because its evidence rule cost more calls than it had. Teaching it to plan its time did not help; giving it the calls did.

For an enterprise, the rule comes down to two questions. Can one agent reach the work before the deadline, at the tool calls it may make per step? If not, and the tools allow it, give it more calls before reaching for a team. Does the evidence fit in one agent’s context? If not, a team that keeps its facts in a shared record holds up where one agent breaks. When both answers are yes, train one agent: a team only adds cost. When you do use a team, give it someone whose only job is to compose the answer.

What this does not show yet:

One generated company and one model, ten reviews in the main comparison. The untrained comparison and the transfer checks rest on one run per review.
The held-out reviews are new, but drawn from the same kinds of cause as training. Our split with unseen kinds (currency translation, supply cascades, data-pipeline lags) has not been run.
Training was short: two update steps per arm. The single agent was never retrained at its new capacity, and the team was never given more calls per member; either could move the curve. The single agent’s lead at nine calls is at the edge of significance (p = 0.051); its recovery with more calls is not (p = 0.002 at every point).
The context limit was simulated by shrinking the budget on small evidence, and real long contexts degrade more gradually than a hard cut. Our harness drops the oldest results first; one that pinned the portfolio summary, or gave the single agent a persistent draft outside its context (which is what the team’s board is), might narrow that gap.
The reward scores the final review. Scoring intermediate steps, which we have designed, is untested.

Three experiments follow directly. A single agent with a record of its own: a pinned summary or a persistent draft outside its context, to see whether one agent can keep its facts the way the team’s board does. Stepwise rewards: potential-based shaping on the grader’s score of the draft as it grows, plus a few verifiable process checks, to see whether it trains faster without opening new ways to game the reward. And the full map: breadth, coupling, deadline, calls per step and context varied together, so the rule above becomes a chart a team can read off for its own job.

If you run reviews like this and are weighing agents for them, we would like to hear from you.

All notes