Research notes
Short-form writing on the techniques we work on and what we learn along the way.
- 2026-09-22
When a team of agents beats one: the weekly performance review
We built an analytics job designed to be too wide for one agent: explain every business unit and region against plan, before a deadline. Trained on the same twelve reviews, with three tool calls per step, a team of seven agents beat the trained single agent on all ten held-out reviews (0.509 against 0.350 reward, p = 0.002). Then we gave the single agent more tool calls per step, and nothing else: with six it drew level, and with nine it led the team (0.577) at a third of the tokens. The team’s advantage was parallel tool capacity, and one agent can have that. What it cannot get from more calls is context: when the evidence outgrew one agent’s window, it broke, inventing most of its numbers, while the team, holding its facts on a shared board, lost 0.15.
- 2026-09-21
Agents that decide for themselves when to team up: two pilots
Three agents explore on their own, talk on a shared channel, and team up only when it would score better. With the same data they found the same causes, and not one of 59 graded offers to team up would have raised the score. With each agent holding one department’s data they talked twice as much, but the vote threw away what the minority found, and training made them worse. One agent trained on two-cause cases beat every team: 80% fully correct on both test sets.
- 2026-09-17
Point it at a warehouse, get an analyst trained on the answer
Root-cause analysis has no training signal: a warehouse records what happened, never why. So we generate a simulated twin of the warehouse where the answer is known exactly, train an investigator there, and send it back to the real data read-only. One measured run: 27 minutes from an unseen database to a trained investigator, 55k tokens per investigation after that.
- 2026-09-16
A swarm of agents, trained on reward: better answers, five times the cost
We replaced the single investigating agent with a team - a lead that plans and assigns, three specialists working in parallel - on the same 20 held-out investigations. Training the team gave the clearest improvement we have measured (60% to 80% fully correct, p = 0.023), but the trained team still did not beat one trained agent on the scoring rubric: more accurate, at 4.9 times the tokens.
- 2026-09-15
Pilot: an agent that learned from reward investigated as well, at 40% of the cost
A pilot in a simulated company: an investigation agent that learned a playbook from environment reward matched the best untrained agent (70% fully correct) with 60% fewer tokens, was best on cause types it never trained on, and learned one rule that went too far. Too small to be significant; the full test plan is below, and results will follow.
- 2026-09-15
Reward alone taught a frozen model the house rules
A frozen gpt-5.4 learned undocumented accounts-payable conventions from environment reward, going from 0% to 98% fully correct on held-out invoices without a single weight update. The first learner reached 26%; why it stalled was the most useful finding.
- 2026-06-16
You can’t trust a patch without a certificate
The concluded study: calibrating the certificate so its verdicts mean something, why low-rank patches land on the edge of certifiability, route-don’t-merge as the settled install policy, and the framework holding up across a 5× model-size range and a second, unrelated domain.
- 2026-06-11
Certified capability repair: what we’ve learned so far
First results from our repair program: honest capability measurement, a certificate that correctly refused a patch that looked great by training loss, and proof that naively composing two good patches destroys both.