RL environments for AI agents
An agent that investigates, decides and acts is judged on the work it produces, not on how closely it echoes an example. The usual ways to make one better — a longer prompt, a library of demonstrations, a fine-tuning set — all train on somebody else's answer. We build the alternative: a reinforcement-learning environment designed around one job, where the agent attempts the real task and is scored on what it actually produced.
Why not just fine-tune?
Fine-tuning is imitation. Its loss rewards the model for predicting the next token of a demonstration, with the correct previous tokens fed back at every step. That is a different task from the one the model performs in production, where it generates freely and has to live with its own mistakes.
Our earlier research measured what that gap costs. Scoring a model with teacher forcing overstated its real capability by an amount that grew almost linearly with how much room it had left to fail — the weaker the model on a task, the more imitation flattered it. And the supervised patches we trained to fix specific failures looked excellent by training loss while quietly damaging capabilities they were never meant to touch. Read the paper.
An environment trains on the model's own attempts instead. The model acts, the result is checked, and it is reinforced toward what actually worked — under the same conditions it will run in for your users.
Drawing out what is already there
The model behind an agent can usually do the task some of the time. For many jobs the gap is not missing knowledge: the model produces the right behaviour occasionally, but does not consistently reach for it. Reward is well suited to closing that gap. It steers the model toward behaviour it can already produce, until that behaviour is what it produces by default.
That is the bet at the centre of Perception: for a well-specified job, an environment gets more out of an agent than a fine-tuning set or a longer prompt built with the same effort — because it trains on outcomes the agent reached itself, rather than on demonstrations it was asked to copy.
What goes into an environment
- Tasks. A distribution of instances drawn from the use case — including the awkward ones — with held-out sets the agent never trains on.
- Tools and interface. Whatever the agent acts through in the real job: an API, a database, a document store, a form.
- A verifier. The reward. It checks what the agent actually produced: the query returns the right rows, the call reaches the right function with valid arguments, the conclusion names the cause that was really there.
- Guards against reward hacking. An agent under optimisation pressure will find shortcuts in any verifier. A reward is treated as untrusted until it has been attacked — reading transcripts against ground truth is how we attack it.
- A certificate. The release gate: task gain measured on held-out instances, and a check that the general abilities the use case depends on did not regress.
How we would work on your use case
- Define it. What the agent should do, what success means, and what it must not break.
- Build the environment. Tasks, tools, verifier and held-out evaluations, stress-tested for reward hacking.
- Train. Against the environment — either by updating the model's weights, or by keeping them frozen and learning a readable playbook from reward, which is how our published results were produced.
- Certify. Measure the gain on held-out work against an untrained baseline, with a regression check across the abilities that matter.
Where we are
Environments are running, and results are published as they come in — including what fails. So far: an agent that learned a playbook from reward matched the best untrained agent at 40% of the tokens; a swarm of agents trained the same way improved significantly on the untrained swarm, but did not beat one trained agent once its cost was counted; and a framework that generates an environment from a real warehouse schema, trains in it, and investigates the real data. Read the notes.