Reward alone taught a frozen model the house rules

Every organisation runs on conventions its documents never state: how a supplier’s name is written in the ledger, what an invoice number looks like, which account a purchase belongs to. A general-purpose model cannot know them, and that is usually where fine-tuning starts.

We asked a narrower question first. If an environment can score the work, how much of that gap closes from reward alone, without touching the model’s weights? On a synthetic accounts-payable task, a frozen gpt-5.4 went from 0% to 98% fully correct extractions on 100 held-out invoices in six learning steps, matching a copy of the same model that was simply handed the rules. The first version of our learner only reached 26%. Why it stalled turned out to be the most useful finding of the study.

Objective

The practical question: can a model learn an organisation’s unwritten conventions from an environment that checks its output, rather than from examples it is trained to copy?

Fine-tuning answers this by collecting labelled examples, training, hosting the result, and repeating the cycle whenever a rule changes. An RL environment answers it differently: let the model attempt the real task, verify what it actually produced, and reinforce what worked. Our longer-term work applies that loop to model weights. This study isolates the environment half — the task, the verifier and the evaluation discipline — and pairs it with the cheapest learner we could build: one that never updates weights at all, and instead writes what it learns into a playbook the model reads.

We set three conditions for calling the result a success, the same discipline we use for certificates:

1. Held-out gain. The improvement has to appear on tasks the learner never saw, under free generation.
2. No silent regression. Every update is checked against fresh tasks before it is kept.
3. A fair frame. The number must sit between a no-learning control, which measures noise, and an oracle, which measures the ceiling.

Plan

An environment with rules nobody wrote down

The task is invoice extraction. Each episode shows the model one supplier document and asks for seven fields as JSON: vendor, invoice number, date, total, currency, payment terms and expense category. The request names the fields and nothing else — deliberately under-specified, the way real business requests are.

The reference records follow house conventions the model is never told:

• Vendor: upper case, with the legal suffix removed (Acme Cloud Services, Inc. → ACME CLOUD SERVICES).
• Invoice number: INV- followed by the number without leading zeros.
• Date: ISO 8601, and numeric dates on supplier documents are day-first.
• Total: tax included, computed when a document only gives a subtotal and a VAT rate.
• Currency: the ISO 4217 code, never the symbol.
• Payment terms: whole days (Payable within two weeks → 14).
• Category: one of five ledger labels, chosen by what was bought, not who sold it.

The documents are generated, which is what makes the study clean: we control the ground truth exactly, so the ceiling is known and every miss is attributable. They vary layout, date formats, currency symbols, number padding, legal suffixes and whether the total is stated or has to be computed, and every document names the buyer (Contoso) as a distractor.

A verifier that checks the record, not the prose

Reward is computed field by field: each of the seven fields that exactly matches the house record earns 1/7. An episode counts as fully correct only if all seven match. There is no partial credit for a plausible-looking value — Office Supplies is not office_supplies. That strictness is the point: it is what a downstream accounting system would enforce.

A learner that changes words, not weights

The policy is the frozen model plus a playbook: a short list of lessons placed in its instructions. Each learning step runs this loop (Figure 1):

1. Sixteen training invoices, four attempts each.
2. The verifier scores every attempt.
3. Reflection: for each invoice where attempts scored differently — or all failed — the model compares the best attempt with the worst, reads the reference record the way one reads a grader’s feedback, and proposes general lessons.
4. A curator merges them into the playbook with explicit add, edit and delete operations, capped at 20 lessons so the prompt can’t grow without bound.
5. A gate runs the old and new playbooks on 16 fresh training invoices. The update is kept only if the score doesn’t drop.

If this resembles GRPO, that is intentional. Sampling a group of attempts per task and learning from how they differ is the same group-relative idea; here the “advantage” is written in language and applied to a playbook instead of a gradient.

Controls that make the number mean something

• Held-out split: 100 invoices scored before and after learning, never shown to the learner.
• Control arm: the unchanged model evaluated twice, which measures run-to-run noise.
• Oracle arm: the model given the written conventions and the full category mapping, which measures the ceiling.
• Statistics: per-invoice paired differences, a bootstrap 95% confidence interval and a sign-flip permutation test.

Experiments

Control and ceiling

Evaluated twice, the unchanged model scored 0.583 both times (95% interval of the difference −0.010 to +0.011). Anything larger than about a hundredth is signal. It already read amounts (99%) and payment terms (100%) almost perfectly, and failed exactly the convention fields: vendor 0%, category 19%, invoice number 21%. Not one of the 100 invoices was fully correct.

Given the rules in its prompt, the same model scored 0.994, with 96% fully correct. So the gap is conventions, not reading ability, and the ceiling is high.

Reflection v1: real gains, then a plateau

The first learner made fast, real progress. Within two steps it learned the vendor convention (0% → 100%) and date and currency formats, and the gate rejected two updates that would have made things worse. Then training reward flattened at about 0.73. Invoice numbers and categories barely moved, and only 3 of 384 training attempts were fully correct (Figure 3, grey).

The diagnosis: lessons that read well but can’t be followed

We read the training attempts mid-run, before the held-out result existed. Two patterns stood out. Categories were semantically right but formatted wrong: Hardware, Office Supplies. Invoice numbers had their leading zeros stripped but no INV- prefix.

The playbook explained why. The reflector had seen the reference records, yet its lessons were hedged: strip leading zeros “only when examples indicate they are non-semantic”; apply the INV- prefix “when examples or task patterns support that canonical format”. Our reflection prompt had asked for “general rules, not facts about this task”, and the reflector obliged with abstractions conditioned on examples. But the model being taught never sees examples or references — it sees one invoice and its instructions. The lessons sounded thorough and gave it nothing to act on.

It echoes our earlier finding that a signal can look excellent by one measure and still mislead. Here the lessons read like expertise; only the verifier showed they weren’t working.

Reflection v2: one change

We changed only the reflection and curation instructions. They now state that the agent sees nothing but the task, forbid conditional hedges, and ask for conventions as direct rules with exact formats and example values. Facts about a single invoice remain excluded. Configuration, seed and held-out set stayed the same.

The first update alone lifted gate reward from 0.607 to 0.929. By the third step, all 64 training attempts were fully correct, so there was nothing left to contrast, and the remaining steps made only small category refinements.

Outcomes

Both learners’ improvements are significant (p = 0.0002, the smallest value 5,000 permutations can report). Reflection v2 improved all 100 held-out invoices and made none worse.

The per-field view shows where the gap lived and that closing it cost nothing elsewhere: the fields the model already handled — total amount and payment terms — stayed at 98–100%. On the invoice from Figure 2, the same model, given the same request, went from two correct fields to seven:

What remains wrong. Two of the 100 invoices still miss one field, both the total. One is a $1.00 arithmetic slip. The other is a one-cent difference caused by rounding order: the learned rule rounds subtotal × (1 + rate), while the house rounds the tax line before adding it. That is a genuine convention the six steps did not surface.

What the learned artifact is. The whole result of learning is a nine-lesson playbook in a JSON file, such as “Normalize invoice_number to INV-<number> … Example: Invoice #0001234 → INV-1234“. It can be read, reviewed, diffed and reverted like any configuration change, and swapped back out instantly. It isn’t perfect: the category lesson lists labels that don’t exist in this ledger (shipping, utilities), a sign that the taxonomy was inferred rather than known.

What it cost. The full v2 run — baseline evaluation, 384 training attempts, gate checks, reflection and final evaluation — used about 450k tokens, once. The recurring cost is the playbook itself: about 590 extra prompt tokens per request (156 → 746). Average latency barely moved (2.24 s → 2.30 s per request), within the run-to-run variation the control showed.

Conclusion

What this shows. For a gap made of conventions, an environment with an exact verifier was enough to close it from reward, with no weight updates, and the gain held on invoices the learner never saw — at the level of a model that was simply told the rules. Two further lessons matter as much as the headline:

• The learner’s design mattered as much as the loop. The same environment, model and budget produced 26% or 98% depending on whether lessons were hedged or concrete. Lessons have to be written for what the policy can actually see.
• The discipline did real work. The gate blocked two regressions, the control showed the noise was about a hundredth, and the oracle gave the result a meaning. Without the verifier, v1’s playbook would have looked like success.

What it doesn’t show yet.

• It is one seed on one model.
• Reflection v2 was evaluated on the same 100 held-out invoices as v1. The change was designed from training attempts and committed before v1’s held-out result arrived, but a fresh held-out set still has to confirm it.
• Reflection read the reference records, like a grader’s feedback. Learning from the reward score alone is the stricter test.
• The environment is synthetic and its conventions are perfectly consistent; real documents are not.
• One round of learner redesign was done by people, and that effort belongs in the cost.

Where this goes next. This was the weight-free half of the question. The same environment and verifier now serve as the training signal for the weight-based half: GRPO on an open base model, scored by exactly the same checks, so the two approaches can be compared directly. Before that, we will replicate v2 across three seeds on a fresh held-out set, run reward-only reflection, repeat on a smaller model, and move from synthetic invoices to a real use case.

If your team has a task governed by conventions like these, bring it to us.

All notes