Research notes

Short-form writing on the techniques we work on and what we learn along the way.

• 2026-09-17

  Point it at a warehouse, get an analyst trained on the answer

  Root-cause analysis has no training signal: a warehouse records what happened, never why. So we generate a simulated twin of the warehouse where the answer is known exactly, train an investigator there, and send it back to the real data read-only. One measured run: 27 minutes from an unseen database to a trained investigator, 55k tokens per investigation after that.

• 2026-09-16

  A swarm of agents, trained on reward: better answers, five times the cost

  We replaced the single investigating agent with a team - a lead that plans and assigns, three specialists working in parallel - on the same 20 held-out investigations. Training the team gave the clearest improvement we have measured (60% to 80% fully correct, p = 0.023), but the trained team still did not beat one trained agent on the scoring rubric: more accurate, at 4.9 times the tokens.

• 2026-09-15

  Pilot: an agent that learned from reward investigated as well, at 40% of the cost

  A pilot in a simulated company: an investigation agent that learned a playbook from environment reward matched the best untrained agent (70% fully correct) with 60% fewer tokens, was best on cause types it never trained on, and learned one rule that went too far. Too small to be significant; the full test plan is below, and results will follow.

• 2026-09-15

  Reward alone taught a frozen model the house rules

  A frozen gpt-5.4 learned undocumented accounts-payable conventions from environment reward, going from 0% to 98% fully correct on held-out invoices without a single weight update. The first learner reached 26%; why it stalled was the most useful finding.

• 2026-06-16

  You can’t trust a patch without a certificate

  The concluded study: calibrating the certificate so its verdicts mean something, why low-rank patches land on the edge of certifiability, route-don’t-merge as the settled install policy, and the framework holding up across a 5× model-size range and a second, unrelated domain.

• 2026-06-11

  Certified capability repair: what we’ve learned so far

  First results from our repair program: honest capability measurement, a certificate that correctly refused a patch that looked great by training loss, and proof that naively composing two good patches destroys both.