# Perception — full text > Full text of every page on https://perception.club, for AI systems and researchers. > Summary: https://perception.club/llms.txt ## Perception — Applied AI research URL: https://perception.club/ Applied AI research # RL environments for AI agents. An agent that investigates, decides and acts does not get better by copying examples of the work. We build the environment it practises in — the tasks, the tools it acts through, and a verifier that scores what it actually produced — and we measure the gain on work it has never seen. Bring us a use caseHow environments workRL environmentsHow they workWhat we have builtAgents measured on held-out workEarlier researchModel repair and certificationBring us a use caseStart a conversationWhy environments ## Agents get better by doing the work, not by reading it. A prompt library and a set of examples tell an agent what good work looks like. An environment makes it do the work: attempt the real task, act through real tools, and be scored on what it actually produced. What improves is the behaviour your users see, because it is the behaviour that was rewarded. ### Experience, not imitation Fine-tuning optimises a model to match examples. An environment optimises for the outcome: did the agent actually solve the task, choosing its own steps as it does in production? Why not fine-tuning ### Draw out what is already there The model behind your agent can usually do the task some of the time. Reward steers it toward the version of itself that does it reliably, instead of overwriting it with a narrow set of demonstrations. The bet ### Verified, not assumed Every environment ends in a certificate: the task improved on held-out instances, and nothing the use case depends on quietly got worse. We publish what that discipline catches, including our own mistakes. Earlier researchHow it works ## An environment built around the job your agent does. We start from what success looks like for your task, and what it costs to get it wrong. Around that we build the tasks, the tools the agent acts through, and a verifier that checks what it actually produced. The agent trains against that environment — by updating weights, or by keeping a readable playbook with the weights frozen — and the result is certified: gain on held-out tasks, and no quiet regression in what your use case depends on. See the processEarlier research ## Why we moved to environments. Our first research program repaired trained models with small supervised patches, and measured carefully where that approach breaks. Patches that looked perfect by training loss leaked into neighbouring capabilities. Patches merged together destroyed each other. Imitation signals overstated capability exactly where models were weakest. Those findings are the case for training on outcomes instead. ### When does an LLM help a GNN on relational data? A paired-baseline study of graph-conditioned LLMs: they tie but don’t beat a well-tuned GNN, while a structured GNN encoding beats flat-text RAG by ~50× on token cost. The GNN is the predictor; the LLM is the interface. Read the paper ### You cannot trust a patch without a certificate The unit of trust in model repair is the certificate, not the patch: a patch that looks perfect by training loss can leak, merging two good patches destroys both, and the safe install is to route, not merge — verified across two model scales and a second domain. Read the paperAll earlier researchNotes ## Recent writing Short-form write-ups of what we find, published as the work concludes. - 2026-09-17 ### Point it at a warehouse, get an analyst trained on the answer Root-cause analysis has no training signal: a warehouse records what happened, never why. So we generate a simulated twin of the warehouse where the answer is known exactly, train an investigator there, and send it back to the real data read-only. One measured run: 27 minutes from an unseen database to a trained investigator, 55k tokens per investigation after that. Read the note - 2026-09-16 ### A swarm of agents, trained on reward: better answers, five times the cost We replaced the single investigating agent with a team - a lead that plans and assigns, three specialists working in parallel - on the same 20 held-out investigations. Training the team gave the clearest improvement we have measured (60% to 80% fully correct, p = 0.023), but the trained team still did not beat one trained agent on the scoring rubric: more accurate, at 4.9 times the tokens. Read the note - 2026-09-15 ### Pilot: an agent that learned from reward investigated as well, at 40% of the cost A pilot in a simulated company: an investigation agent that learned a playbook from environment reward matched the best untrained agent (70% fully correct) with 60% fewer tokens, was best on cause types it never trained on, and learned one rule that went too far. Too small to be significant; the full test plan is below, and results will follow. Read the note All notesWork with us ## Bring us a use case. Tell us the job you want an agent to do well, and what doing it well means. We will tell you plainly whether an RL environment is the right tool, and what it would take to build one. Start a conversation --- ## RL environments for AI agents — Perception URL: https://perception.club/environments RL environments # RL environments for AI agents An agent that investigates, decides and acts is judged on the work it produces, not on how closely it echoes an example. The usual ways to make one better — a longer prompt, a library of demonstrations, a fine-tuning set — all train on somebody else's answer. We build the alternative: a reinforcement-learning environment designed around one job, where the agent attempts the real task and is scored on what it actually produced. ## Why not just fine-tune? Fine-tuning is imitation. Its loss rewards the model for predicting the next token of a demonstration, with the correct previous tokens fed back at every step. That is a different task from the one the model performs in production, where it generates freely and has to live with its own mistakes. Our earlier research measured what that gap costs. Scoring a model with teacher forcing overstated its real capability by an amount that grew almost linearly with how much room it had left to fail — the weaker the model on a task, the more imitation flattered it. And the supervised patches we trained to fix specific failures looked excellent by training loss while quietly damaging capabilities they were never meant to touch. Read the paper. An environment trains on the model's own attempts instead. The model acts, the result is checked, and it is reinforced toward what actually worked — under the same conditions it will run in for your users. ## Drawing out what is already there The model behind an agent can usually do the task some of the time. For many jobs the gap is not missing knowledge: the model produces the right behaviour occasionally, but does not consistently reach for it. Reward is well suited to closing that gap. It steers the model toward behaviour it can already produce, until that behaviour is what it produces by default. That is the bet at the centre of Perception: for a well-specified job, an environment gets more out of an agent than a fine-tuning set or a longer prompt built with the same effort — because it trains on outcomes the agent reached itself, rather than on demonstrations it was asked to copy. ## What goes into an environment - Tasks. A distribution of instances drawn from the use case — including the awkward ones — with held-out sets the agent never trains on. - Tools and interface. Whatever the agent acts through in the real job: an API, a database, a document store, a form. - A verifier. The reward. It checks what the agent actually produced: the query returns the right rows, the call reaches the right function with valid arguments, the conclusion names the cause that was really there. - Guards against reward hacking. An agent under optimisation pressure will find shortcuts in any verifier. A reward is treated as untrusted until it has been attacked — reading transcripts against ground truth is how we attack it. - A certificate. The release gate: task gain measured on held-out instances, and a check that the general abilities the use case depends on did not regress. ## How we would work on your use case - Define it. What the agent should do, what success means, and what it must not break. - Build the environment. Tasks, tools, verifier and held-out evaluations, stress-tested for reward hacking. - Train. Against the environment — either by updating the model's weights, or by keeping them frozen and learning a readable playbook from reward, which is how our published results were produced. - Certify. Measure the gain on held-out work against an untrained baseline, with a regression check across the abilities that matter. ## Where we are Environments are running, and results are published as they come in — including what fails. So far: an agent that learned a playbook from reward matched the best untrained agent at 40% of the tokens; a swarm of agents trained the same way improved significantly on the untrained swarm, but did not beat one trained agent once its cost was counted; and a framework that generates an environment from a real warehouse schema, trains in it, and investigates the real data. Read the notes. Bring us a use case --- ## Earlier research — Perception URL: https://perception.club/research Earlier research # Can a trained model be fixed without retraining it? This was Perception's first research program, and the papers and notes below remain its record. What we learned — that supervised patches leak into capabilities they were never meant to touch, fail when combined, and look better by training loss than they are — is why our central work is now RL environments. That was the research question at the centre of Perception's first program. Training a large model costs real money and real energy. When it doesn't behave the way we need, the default answer is to retrain — or to discard the model entirely. We're working on a third option: surgical edits to the weights and representations of models that already exist. The bet is that the cost a team has already paid for pre-training or fine-tuning can be recovered through precise interventions, rather than written off. If true, it changes what "alignment" means in practice — alignment as repair, not alignment as restart. ## The program — certified capability repair The program treated repair as a lifecycle, not a one-off edit. An observed failure is classified; a capability is selected against strict eligibility criteria (it must be measurable, repeatable, isolable, and benchmarkable); a patch is generated; and the patch is judged by a certificate — a per-dimension, pass/fail measurement of whether the target capability actually improved, whether the declared neighborhood stayed put, whether general ability regressed, and what the patch cost. Patches that will be installed together additionally need a composition proof: evidence they survive co-installation. The certificate, not the patch, is the product. A patch is a claim; the certificate is the evidence. The eventual system contract is simple to state — input: failure examples; output: a certified patch. The first study is concluded and written up as a working paper. Along the way the certificate refused a patch that looked perfect by training loss; we calibrated its thresholds from a measured noise floor so the verdicts mean something; we found that low-rank patches reach the edge of certifiability but land on it; and we settled how to install two patches at once — the answer is to route between them, not merge their weights, because naive merging drops both capabilities below the unpatched model. The findings hold up under pressure: the measuring-stick result is a near-deterministic law across a 5× range of model sizes, and the whole framework transfers to a second, unrelated domain (tool-calling) on an external benchmark. Read the paper — You Cannot Trust a Patch Without a Certificate · PDF Shorter write-ups of the findings: - You can't trust a patch without a certificate — the concluded study: calibration, the patch-mechanism wall, and route-don't-merge. - Certified capability repair: what we've learned so far — the first results: honest measurement, a leaky patch caught, composition collapse. ## Publications Working papers, released as they conclude. - 2026-06-19 ### When does an LLM help a GNN on relational data? A paired-baseline study of graph-conditioned LLMs: they tie but don’t beat a well-tuned GNN, while a structured GNN encoding beats flat-text RAG by ~50× on token cost. The GNN is the predictor; the LLM is the interface. Read · PDF - 2026-06-16 ### You cannot trust a patch without a certificate The unit of trust in model repair is the certificate, not the patch: a patch that looks perfect by training loss can leak, merging two good patches destroys both, and the safe install is to route, not merge — verified across two model scales and a second domain. Read · PDF ## Techniques we worked with ### Causal tracing Before you can edit a behaviour, you have to find it. Causal tracing identifies which layers and components in a model are actually responsible for a given output. The goal is precision — knowing where to intervene without disturbing what is already working. ### Rank-one model editing (ROME) ROME is a technique for updating specific factual or behavioural associations in a model with a minimal, targeted weight change. We treat it as a foundational primitive: small, verifiable, reversible edits — the opposite of full retraining. ### Task vectors and weight arithmetic Fine-tuning produces a direction in weight space. Task vectors let us add, subtract, and compose those directions across models. This opens up a question we care about: can you remove a learned behaviour as cleanly as you added it? ### Representation engineering (RepE) Some behaviours don't live in weights — they live in the activations as the model runs. Representation engineering operates on those activations directly, steering behaviour at inference time. It complements weight editing and gives us a second control surface. ## What we are not We are not a foundation-model lab. We do not train models from scratch. Our research starts from the assumption that the model already exists, the training run already happened, and the question is what to do next. ## The applied bet Editing a model precisely is only useful if you know what better looks like. Most "alignment" work answers this with proxies — benchmarks, preference data, rule sets. We think the harder and more honest answer is longitudinal evidence of how an individual actually develops over time. That is what Sage is building, starting with childhood. The two threads — precision editing and longitudinal evidence — meet in the same question: how do you make AI useful to a specific human, not a generic one, without retraining from zero for each person? What we work on now: RL environments --- ## When does an LLM help a GNN on relational data? — Perception URL: https://perception.club/research/when-does-an-llm-help-a-gnn # When Does a Language Model Help a Graph Neural Network on Relational Data? A Rigorous Study of Graph-Conditioned LLMs and a Portable GNN-for-RAG Architecture *Sunil Kumar, Perception. Contact: contact@perception.club.* --- ## Abstract Relational deep learning (RDL) turns a multi-table database into a temporal heterogeneous graph and learns directly over it with a graph neural network (GNN), avoiding manual feature engineering. A recent line of work (e.g., Rel-LLM) couples such a GNN to a large language model (LLM) by injecting a *graph soft-prompt* into the model, claiming improved predictive accuracy and a path to natural-language interaction. We conduct a careful, paired-baseline study of this claim on the RelBench `rel-f1`/`driver-dnf` entity-classification task. Our central methodological finding is that **the apparent gains of graph-conditioned LLMs are highly sensitive to the strength of the GNN baseline**: when compared against a *well-tuned, paired, best-validation* GNN, every faithful LLM configuration we tested — decoupled vs. joint training, 1.5B vs. 8B models, and single- vs. nested multi-token graph prompts — **ties but does not beat** the GNN on prediction (joint 8B nested prompt: 0.7327 ± 0.010 AUROC vs. GNN 0.7286 ± 0.010; Δ = +0.004, within run-to-run noise). Earlier "wins" we observed (+0.025 to +0.037 AUROC) were artifacts of an under-tuned or cross-run baseline. Conversely, when we compare the GNN encoding against the classical alternative — **flat-text retrieval-augmented generation (RAG)** that serializes the same neighborhood into the prompt — the GNN wins decisively on *both* accuracy (≈0.73 vs. ≈0.34 zero-shot) and *context efficiency* (≈3–17 vs. ≈870 tokens per entity, ≈50×). We reconcile these results with a deployment-oriented architecture: the GNN is the *predictor*; the LLM's value is *retrieval, natural-language interaction, and explanation*. Because the soft-prompt technique is white-box only, we describe a **portable GNN-for-RAG** design in which the GNN acts as a model-agnostic *tool* whose structured output can be consumed by open-weight or frontier LLMs alike. The code, per-run logs, and an implementation manual are available on request. **Keywords:** relational deep learning, graph neural networks, large language models, retrieval-augmented generation, tabular/relational ML, RelBench. --- ## 1. Introduction Most enterprise data lives in relational databases — many tables linked by primary- and foreign-key (PK/FK) relationships, with per-row timestamps. The dominant predictive workflow flattens this structure into a single feature table by hand, discarding relational and temporal signal and incurring large engineering cost. **Relational deep learning (RDL)** [Fey et al., 2024] reframes the database as a *relational entity graph* and learns over it end-to-end with a GNN, and **RelBench** [Robinson et al., 2024] standardizes this as a benchmark across classification, regression, and recommendation tasks. A natural and appealing idea is to combine the GNN with an LLM. The LLM brings world knowledge, natural-language interaction, and in-context reasoning; the GNN brings an efficient, leakage-free encoding of the relevant subgraph. **Rel-LLM** [Wu et al., 2025] realizes this by projecting GNN node embeddings into the LLM's embedding space as a *graph soft-prompt* (via `inputs_embeds`) and fine-tuning with LoRA, reporting accuracy improvements over a GNN baseline. This raises two questions that motivate our study: 1. **Does conditioning an LLM on a GNN actually improve predictive accuracy over a strong GNN?** 2. **If we instead feed the same relational neighborhood to the LLM as plain text (classical RAG), how does a structured GNN encoding compare on accuracy and on token cost?** We answer both with a deliberately conservative experimental protocol. Our key insight is that **the answer to (1) depends almost entirely on how well the GNN baseline is tuned and how fairly it is compared.** Using a *paired, same-seed, best-validation* GNN head as the baseline — rather than a cross-run or under-tuned number — we find that graph-conditioned LLMs tie but do not surpass the GNN, even at 8B scale with the full nested-prompt recipe. For (2), the GNN's structured encoding is both far more accurate and ≈50× more token-efficient than zero-shot flat-text RAG. These results are individually unsurprising in hindsight but, taken together, sharpen the design space: **for relational *prediction*, the GNN is the engine; the LLM earns its place through *retrieval, language, and explanation*, not through extra predictive accuracy.** Because the soft-prompt mechanism cannot attach to a frontier API model (which accepts text, not embeddings) and is not even portable across open-weight models (the projection is per-LLM), we argue for a **portable GNN-for-RAG** architecture in which the GNN is a standalone, model-agnostic *tool*. We detail this design and accompany it with a practitioner manual. **Contributions.** - A **paired-baseline methodology** for evaluating graph-conditioned LLMs, and evidence that prior positive results are fragile to baseline strength (Section 5.1). We document a concrete false-positive (+0.037 AUROC) that vanished under a fair baseline. - A **systematic sweep** over decoupled/joint training, 1.5B/8B model scale, and single/nested graph prompts, all of which **tie** a well-tuned GNN on `rel-f1`/`driver-dnf` (Section 5.1). - A **GNN-vs-RAG comparison** isolating the value of structured graph encoding over flat-text serialization on both accuracy and token efficiency (Section 5.2). - A **portable GNN-for-RAG architecture** that reconciles these findings for deployment with open-weight and frontier LLMs, plus an open implementation (Section 6, Appendix). --- ## 2. Related Work **Relational deep learning and RelBench.** RDL [Fey et al., 2024] casts a database as a temporal heterogeneous graph (the *relational entity graph*) and learns task-specific predictions with message-passing GNNs over leakage-free temporal neighbor samples. RelBench [Robinson et al., 2024] provides datasets and standardized tasks; the reference GNN combines column encoders [PyTorch-Frame; Hu et al., 2024], a heterogeneous GraphSAGE backbone [Hamilton et al., 2017], and per-task heads. **RelGNN** [Yuan et al., 2025] introduces atomic-route composite message passing for state-of-the-art classification/regression, and **ContextGNN** [Fey et al., 2024b] targets recommendation with a hybrid pairwise/two-tower head. **LLMs over tabular and relational data.** A body of work serializes rows/tables into text for in-context learning, which is sensitive to prompt design and context length. **Rel-LLM** [Wu et al., 2025] instead injects GNN embeddings into a frozen LLM as a soft-prompt with LoRA adaptation, reporting gains over a GNN baseline — the primary claim our study scrutinizes. **Parameter-efficient adaptation and quantization.** LoRA [Hu et al., 2021] and 4-bit QLoRA [Dettmers et al., 2023] make LLM fine-tuning feasible on a single GPU; we use 4-bit NF4 inference/adaptation throughout. **Retrieval-augmented generation.** RAG [Lewis et al., 2020] augments an LLM with retrieved context. Our "text-RAG" baseline is the relational instantiation: retrieve an entity's temporally-valid neighborhood, serialize to text, and prompt the LLM. This is the model-agnostic alternative to the soft-prompt, and the comparison isolates the effect of *structured* (GNN) vs. *flat-text* encoding. --- ## 3. Method ### 3.1 Database → Relational Entity Graph (REG) Given tables with declared PK/FK constraints and per-row timestamps, we construct a heterogeneous graph: each row is a typed node; each PK→FK reference is a typed edge. **Bridge/junction tables** (rows with ≥2 FKs) are detected and used to form single-hop cross-entity connections ("atomic routes"). For a prediction about entity *e* at *seed time* *t*, temporal neighbor sampling admits only rows with timestamp < *t*, preventing label leakage. Column values are encoded per-type by a column encoder, yielding initial node features. ### 3.2 GNN core (the predictor) The GNN core ("RelCore") is a heterogeneous message-passing encoder over sampled temporal neighborhoods, with a task head (binary classification / regression / link prediction). Our backbone follows the RDL reference (heterogeneous GraphSAGE with temporal encoding); we additionally implement an **atomic-route composite** variant (RelGNN-style) with a learned bridge-routing gate. Training uses focal loss [Lin et al., 2017] for imbalanced binary tasks; model selection is by **best validation** AUROC. This core is the system's *predictor* and is the strong baseline that all LLM variants are measured against. ### 3.3 Two ways to attach an LLM We study two mechanisms for letting an LLM use the same relational neighborhood. **(a) Graph soft-prompt (white-box; "RelLM").** GNN embeddings are projected into the LLM's token-embedding space and prepended as virtual tokens via `inputs_embeds`; a LoRA-adapted, frozen LLM reads a Yes/No answer from the next-token distribution. We evaluate: - *Decoupled*: GNN frozen, embeddings precomputed, only the LLM side trains. - *Joint*: gradients flow GNN↔LLM end-to-end (the paper's recipe), with gradient accumulation to raise the effective GNN batch under the LLM's small physical batch. - *Single vs. nested prompt*: one pooled graph token, vs. a nested multi-token prompt (a seed token plus one token per incoming relation, each the mean of neighbors along that relation). This mechanism is **white-box only**: it requires access to the LLM's embedding matrix and hidden states, so it cannot target a frontier API and is not portable across open-weight models (the projection is per-LLM). **(b) Flat-text RAG (model-agnostic).** The entity and its temporally-valid neighbors are serialized to JSON-like text, placed in a prompt with a task question, and the LLM's Yes/No logits are read (zero-shot). This is *model-agnostic*: the same prompt shape ports to any causal LM and to frontier APIs. It is the classical RAG baseline against which the GNN's structured encoding is measured. ### 3.4 Evaluation protocol (the crux) Comparisons between an LLM variant and "the GNN" are only meaningful if the GNN baseline is (i) **well tuned**, (ii) **paired** — trained on the *same* embeddings/seed where applicable — and (iii) selected by **best validation**, not final epoch. We additionally report **multi-seed mean ± std**, because run-to-run variance on this task is ≈±0.01 AUROC, large enough to manufacture spurious single-run "wins." Section 5.1 shows that relaxing any of these conditions produces a false positive. --- ## 4. Experimental Setup **Task.** RelBench `rel-f1` / `driver-dnf`: predict whether a driver will record a did-not-finish (DNF) in the near future (binary classification; positive = DNF, a minority class). Metric: test AUROC. We chose this task as small but knowledge-rich (Formula 1 entities are well represented in LLM pretraining), which is the most favorable setting for an LLM to help. **Models.** GNN core as in Section 3.2. LLMs: `Qwen2.5-1.5B-Instruct` (fp16) and `Qwen3-8B` (4-bit NF4) via HuggingFace Transformers + PEFT (LoRA). For classification we read the final-token Yes/No logits (`logits_to_keep=1`, computing only the last position). **Hardware.** A single NVIDIA T4 (16 GB) for the 1.5B inference and a single L4 (24 GB) for 8B joint training; both via cloud VMs. GNN training fits comfortably on a single 16 GB GPU. **Baseline protocol.** Paired, best-validation GNN head; 3 seeds (42, 1, 2); mean ± std reported. Text-RAG is evaluated zero-shot on a 1000-row sample of val/test. --- ## 5. Results ### 5.1 Graph-conditioned LLMs tie, but do not beat, a well-tuned GNN **Table 1.** Entity prediction on `rel-f1`/`driver-dnf` (test AUROC). All LLM rows use the graph soft-prompt ("RelLM"). | Method | AUROC (mean ± std) | Note | |---|---|---| | Published RDL (reference) | 0.726 | [Robinson et al., 2024] | | **GNN (ours, well-tuned, best-val)** | **0.729 ± 0.010** | the bar | | RelCore composite routing (RelGNN-style) | ≈ GNN (gate → 0) | no significant gain | | RelLM decoupled (1.5B / 8B) | tie / slightly behind | opaque frozen embedding | | RelLM joint, 1.5B, single token | 0.7251 | tie | | RelLM joint, 8B, single token | 0.7285 | tie (+0.004 vs. fair GNN) | | **RelLM joint, 8B, nested prompt (full recipe)** | **0.7327 ± 0.010** | **tie (+0.004, within noise)** | The strongest LLM configuration — joint training, 8B model, nested multi-token graph prompt — reaches 0.7327 ± 0.010 vs. the paired GNN's 0.7286 ± 0.010, a difference of **+0.004 AUROC**, well within the ±0.01 run-to-run noise. The **per-seed deltas straddle zero** (−0.009, −0.005, +0.027): the single positive seed reflects a low-variance draw of the GNN baseline rather than a systematic LLM advantage. **A documented false positive.** In an earlier iteration we observed RelLM "beating" the GNN by +0.025 to +0.037 AUROC. Investigation showed these were **baseline artifacts**: (i) a *cross-run* GNN number rather than a same-seed paired head, and (ii) a GNN baseline selected at the final epoch under a small-batch configuration rather than best-validation under the canonical configuration. Re-running with the paired best-validation GNN erased the gain. We report this explicitly because it is, we believe, the central practical hazard in this subfield: **the measured benefit of a graph-conditioned LLM is dominated by how the GNN baseline is constructed.** **Recommendation track (supporting evidence).** On a recommendation task (`rel-trial`/`condition-sponsor-run`), a text-free two-tower pipeline reached MAP ≈ 0.015 vs. a published text-on ≈0.114; the head choice barely mattered (published two-tower 11.36 ≈ ID-GNN 11.48). The lever on these datasets is **features/text, not GNN architecture** — consistent with Table 1. ### 5.2 The GNN decisively beats flat-text RAG on accuracy *and* token cost **Table 2.** GNN encoding vs. flat-text RAG into the *same* 1.5B-class LLM, on `rel-f1`/`driver-dnf`. | Approach | AUROC | tokens / entity | |---|---|---| | GNN (no LLM) | **0.729** | — | | RelLM (GNN soft-prompt → LLM) | 0.725 | **≈3–17** | | Text-RAG (flat serialized records → LLM, zero-shot) | **≈0.34** (degenerate, f1 = 0) | **≈870** | Zero-shot 1.5B text-RAG is **non-discriminative** (AUROC ≈0.34, below chance, with f1 = 0 indicating near-constant predictions), and was robust to prompt framing (three question phrasings yielded 0.30–0.42). Meanwhile it consumes ≈870 prompt tokens per entity, versus ≈3–17 tokens for the GNN encoding — an ≈**50× context-efficiency advantage**. The GNN's structured encoding both extracts the predictive signal that a small zero-shot LLM cannot pull from flat text *and* does so at a tiny fraction of the context cost. **Caveat (fairness).** The text-RAG point is *zero-shot* with a *small* (1.5B) model; the degeneracy partly reflects model capacity. A **fine-tuned** text-RAG (LoRA on serialized text) and/or a **frontier model** (Claude/Gemini/GPT) is the fairer *accuracy* baseline and is the natural next experiment. The **token-efficiency** conclusion, however, is model-independent and stands regardless. (We also identified and fixed an engineering subtlety — right-truncation silently dropping the trailing question on long prompts — which we correct in the released code.) --- ## 6. Discussion: a portable GNN-for-RAG architecture Two facts shape deployment. First, **the LLM does not improve relational prediction** over a well-tuned GNN at the scales we tested (Section 5.1). Second, **structured GNN encoding dominates flat-text RAG** on both accuracy and token cost (Section 5.2). Together they imply a clean division of labor: > **The GNN is the predictor; the LLM is the interface** — for retrieval, natural-language exploration, and explanation — not for squeezing extra accuracy. The soft-prompt realization of "GNN + LLM," however, is white-box only and per-LLM. To make the GNN *portable* — attachable to open-weight *and* frontier models — we cast it as a **model-agnostic tool**: - **GNN core (standalone).** Trained on the enterprise schema and (optionally) business terminology; exposes `predict(entity, seed_time)` and `retrieve(entity, seed_time) → structured context`. - **Open-weight path.** May optionally use the soft-prompt for tightest coupling, *or* the tool path below. - **Frontier path (portable).** The GNN's output is rendered as **compact structured text** or returned as a **tool result** to a frontier LLM via its tool-use API. The LLM orchestrates: it calls the GNN tool for predictions/retrieval and uses its own language ability for the user-facing answer and explanation. This requires no access to model internals and is identical across providers. This is exactly the GNN-vs-RAG axis of Section 5.2, used constructively: the GNN tool returns a ≈10-token structured encoding (a calibrated prediction plus a small set of the most relevant evidence rows) instead of an ≈870-token raw dump, giving the orchestrating LLM a cheaper, higher-signal context. The accompanying implementation manual specifies the tool schema and the agent loop for both open-weight (HuggingFace) and frontier (Anthropic Claude tool-use) backends. --- ## 7. Limitations - **Single pilot task.** Our deepest results are on one task (`rel-f1`/`driver-dnf`). The methodology (paired best-val baseline, multi-seed) generalizes, but absolute conclusions should be re-checked per dataset; tasks where pretraining knowledge is decisive may favor the LLM more. - **Text-RAG accuracy is a floor, not a ceiling.** It is zero-shot + 1.5B; fine-tuned and frontier RAG are unmeasured here and are the right next baselines. - **Scale.** We test up to 8B; we do not rule out gains at much larger scale, though our trend (no gain from 1.5B→8B) is not encouraging for the prediction-accuracy claim specifically. - **Soft-prompt portability.** The white-box soft-prompt is not portable; our portable design uses the tool path, whose end-to-end accuracy with a fine-tuned frontier orchestrator we leave to future work. --- ## 8. Conclusion We studied whether a language model helps a graph neural network on relational prediction, under a deliberately fair, paired, multi-seed protocol. Across decoupled/joint training, 1.5B/8B scale, and single/nested graph prompts, graph-conditioned LLMs **tie but do not beat** a well-tuned GNN — and we show that previously observed "wins" are artifacts of an under-tuned baseline. In contrast, structured GNN encoding **decisively beats** flat-text RAG on both accuracy and token efficiency (≈50×). The practical conclusion is an architecture in which the **GNN is the predictor and the LLM is a model-agnostic interface** for retrieval, language, and explanation. The code, per-run logs, a consolidated results document, and an implementation manual are available on request to support reproduction and deployment. --- ## Reproducibility The installable `relational_llm/` package, notebooks, the cloud-VM recipe, per-run JSON logs, and a consolidated results document are available on request. Engineering details (temporal sampling, focal loss, the `logits_to_keep` last-token-logits optimization, prompt-truncation handling, and cloud-GPU/stack pitfalls) are documented in the project journal and product manual. --- ## References 1. M. Fey, W. Hu, K. Huang, J. E. Lenssen, R. Ying, J. Leskovec, and others. *Relational Deep Learning: Graph Representation Learning on Relational Databases.* ICML Position Paper, 2024. arXiv:2312.04615. 2. J. Robinson, R. Ranjan, W. Hu, K. Huang, J. Zhang, et al. *RelBench: A Benchmark for Deep Learning on Relational Databases.* NeurIPS Datasets and Benchmarks, 2024. arXiv:2407.20060. 3. W. Wu, et al. *Rel-LLM: Large Language Models for Relational Deep Learning* (graph soft-prompt conditioning of LLMs over relational entity graphs). 2025. arXiv:2506.05725. 4. T. Yuan, et al. *RelGNN: Composite Message Passing for Relational Deep Learning.* 2025. arXiv:2502.06784. 5. M. Fey, et al. *ContextGNN: Beyond Two-Tower Recommendation Systems.* 2024. arXiv:2411.19513. 6. W. Hamilton, R. Ying, and J. Leskovec. *Inductive Representation Learning on Large Graphs (GraphSAGE).* NeurIPS, 2017. arXiv:1706.02216. 7. W. Hu, et al. *PyTorch Frame: A Modular Framework for Multi-Modal Tabular Learning.* 2024. arXiv:2404.00776. 8. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. *LoRA: Low-Rank Adaptation of Large Language Models.* ICLR, 2022. arXiv:2106.09685. 9. T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. *QLoRA: Efficient Finetuning of Quantized LLMs.* NeurIPS, 2023. arXiv:2305.14314. 10. P. Lewis, E. Perez, A. Piktus, et al. *Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.* NeurIPS, 2020. arXiv:2005.11401. 11. T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. *Focal Loss for Dense Object Detection.* ICCV, 2017. arXiv:1708.02002. 12. A. Vaswani, et al. *Attention Is All You Need.* NeurIPS, 2017. arXiv:1706.03762. 13. Z. Hu, Y. Dong, K. Wang, and Y. Sun. *Heterogeneous Graph Transformer (HGT).* WWW, 2020. arXiv:2003.01332. 14. Qwen Team. *Qwen2.5 Technical Report.* 2024. arXiv:2412.15115. --- *Correspondence: Sunil Kumar, Perception — contact@perception.club.* --- ## You cannot trust a patch without a certificate — Perception URL: https://perception.club/research/certified-capability-repair # You Cannot Trust a Patch Without a Certificate: Behavioral Certification and Composition Preservation for LLM Capability Repair *Sunil Kumar, Perception. Contact: contact@perception.club. v1 working draft.* > Provenance. Every quantitative claim in this paper is backed by a committed artifact produced > against a single SHA-pinned base model. The complete experiment trail — including pre-registered > success criteria, refuted hypotheses, and our own incorrect predictions — is the append-only > journal kept alongside the code, available on request. Section numbers in brackets (e.g. [J#26]) point to journal entries. --- ## Abstract Practitioners increasingly *repair* large language models with small, parameter-efficient patches — a LoRA adapter that fixes a specific failure mode, an edit that corrects a behavior — and judge those patches by training loss or a single accuracy number. We argue, and demonstrate, that this is unsafe on two distinct axes. First, a patch that looks excellent by training loss can **leak**: it degrades capabilities it was never meant to touch, in ways a target-metric never reveals. Second, even a good patch is **not preserved under composition**: co-installing it with a second patch can destroy both. We make the certificate the product. A *behavioral certificate* is a per-dimension PASS/FAIL record — efficacy, locality, regression, and cost — scored exclusively under greedy free generation, with bootstrap confidence intervals, against a base model pinned by commit hash. We pair it with a *composition-preservation test* that re-certifies each patch after co-installation. On text-to-SQL with Qwen2.5-1.5B- and 7B-Instruct, we report six results. (1) Teacher-forced token agreement overstates free-generation execution accuracy by an amount that is a near-deterministic linear function of the model's free-generation *headroom* — gap = −0.045 + 0.931 · (1 − free-gen accuracy), R² = 0.99, r = 0.993, p < 10⁻⁴ — and the *same* line holds across a 5× model-size range; an error audit shows the inflation is exactly the family of derailments (spurious joins, hallucinated tables) that teacher forcing masks. The gap is largest precisely in the fallible regime where repair happens and vanishes only at task ceiling, so a certificate must score with free generation. (2) A naive join-repair LoRA whose training loss falls cleanly from 0.55 to 0.165 fails its certificate on three dimensions; 72 of 73 of its collateral failures are a single mechanism, the injection of unwanted joins into queries that need none. (3) The locality threshold cannot be set by fiat; we calibrate it from a benign-perturbation noise floor under a rule fixed before data collection, and show this is both necessary (uncalibrated thresholds reject patches on noise) and sufficient (the calibrated certificate is satisfiable). (4) Naive merging of two individually strong patches drops *both* capabilities below the unpatched base model. (5) This composition conflict is predictable from the patch weights — subspace overlap, delta norms, and an interference ratio — with the counter-intuitive sign that *lower* subspace overlap is *more* destructive; the governing mechanism is destructive interference between patches of comparable magnitude. (6) The optimal install operation is therefore **routing, not merging**: keeping patches separate strictly dominates naive merge across every pair we tested. Finally, on a second, unrelated capability and an external benchmark — tool/function-call generation on the Berkeley Function-Calling Leaderboard — the same framework refuses a patch that is simultaneously useless on its target (efficacy break-even) and harmful off it (a 0.094 instruction-following regression), and we find the same collateral victim, instruction-following, degraded by *both* SQL and tool-call training. The certificate framework, the calibration procedure, the conflict analysis, and the full pre-registered experiment record are available on request. --- ## 1. Introduction ### 1.1 From "a patch" to "a failure" The dominant framing of model editing and parameter-efficient fine-tuning begins with a *patch*: a method produces a weight delta, and the delta is evaluated on a benchmark slice. Deployment, however, begins with a *failure*. An operator observes that a model mishandles a class of inputs, classifies the failure, decides whether the underlying capability is even a legitimate repair target, generates a patch, and then must answer the only questions that matter in production: *Did the patch fix the failure without breaking anything else? And will it survive next to the patches already installed?* These two questions — isolation and composition — are precisely the ones a single benchmark number cannot answer. Our thesis is that the unit of trust in capability repair is not the patch but the **certificate**: a structured, per-dimension, statistically honest statement of what the patch does and does not do, together with a proof that the statement survives composition. The patch is an artifact; the certificate is the product. ### 1.2 Why the obvious evaluation fails Two failure modes of naive evaluation motivate the entire paper. **Inflated measurement.** Capability is often scored with teacher forcing (feeding the gold continuation and measuring per-token agreement) or with lenient string matching. Both overstate the behavior a user actually experiences, because the user gets free, unguided generation. A patch can look strong under teacher forcing and be weak in deployment. A certificate that inherits this inflation certifies a fiction. **Hidden collateral damage and composition collapse.** Even with honest scoring, a patch optimized for one capability can silently move others, and two honest patches can interfere when combined. A target-only metric is blind to both. The literature has begun to notice the first problem — recent work argues editing evaluations dramatically overstate real-world effectiveness and that standard locality metrics are weakly correlated with the strength of the regularizer meant to control them — but no prior framework combines capability- granularity behavioral certificates, composition-preservation testing, and a pre-registered calibration of the very thresholds the verdict depends on. ### 1.3 Contributions 1. **The behavioral certificate (§3).** A per-dimension PASS/FAIL record — efficacy, locality, regression, patch cost — under a non-negotiable evaluation protocol (greedy free generation, set-semantics execution accuracy, bootstrap CIs, SHA-pinned base), with dimensions never averaged. 2. **The measuring-stick law (§5).** A pre-registered, audited demonstration that teacher-forced scoring inflates free-generation capability by an amount that is a near-deterministic linear function of the model's headroom (R² = 0.99), holding across a 5× model-size range, with the error mechanism identified. 3. **A leaky patch caught and explained (§6).** A LoRA whose training loss looks ideal fails its certificate; we localize 72/73 of its collateral failures to one mechanism. 4. **Threshold calibration as a first-class step (§7).** A rule, fixed before data, that derives the locality threshold from a benign-perturbation noise floor — and the finding that this makes the certificate satisfiable where a hand-set threshold does not. 5. **Composition breaks certificates (§9).** Naive merge of two strong patches drops both below base. 6. **Conflict is weight-predictable, and routing dominates merging (§10–§11).** The conflict tracks subspace overlap (with a reversed sign), delta norms, and interference; and the install operation that recovers the lost capability is routing, with naive merge strictly dominated. 7. **The framework transfers to a second domain (§12).** On tool-call generation and an external benchmark (BFCL), the certificate refuses a patch that is both useless on its target and harmful off it, and surfaces a cross-capability effect — instruction-following degraded by both SQL and tool-call training — that an averaged score would hide. ### 1.4 Method as a stance We adopt practices uncommon in empirical deep learning but standard in the experimental sciences. Every threshold and success criterion is **pre-registered** in a dated journal entry before the run that tests it. We record refutations, including a prediction of ours that the data reversed (§10). When a result hinges on a measurement artifact — a probe too small to resolve its own threshold, a floating-point boundary — we say so and fix it *forward only*, never retroactively converting a recorded failure into a pass. The journal is the methods-and-results trail; this paper is its synthesis. --- ## 2. Related Work **Behavioral vs. parameter-space certification.** The closest contemporaneous work, *Provably Safe Model Updates*, certifies updates by constructing invariant regions in parameter space. That guarantee is formal and worst-case but is about parameters; ours is statistical and behavioral, defined directly on greedy generations with confidence intervals at capability granularity. The two are complementary: a parameter-space certificate bounds what *can* change; a behavioral certificate measures what *did* change on the behaviors you declared you care about. **Predicting merge success.** *Demystifying Mergeability*, concurrent with this work, predicts model-merging success from interpretable model properties — essentially the question of our §10. We differ in target and framing: our predictand is *certificate degradation* (does a co-install break a PASS dimension?), and our analysis surfaces a specific mechanism — destructive interference between comparable-magnitude deltas — together with the counter-intuitive result that lower subspace overlap is more harmful. We treat this concurrent work as corroborating, and position our contribution as the certificate-centric and mechanistic account. **Evaluation honesty.** *The Mirage of Model Editing* documents that editing evaluations overstate in-the-wild effectiveness, motivating our insistence on free generation (§5). *Are We Evaluating Edit Locality Properly?* shows existing locality/specificity metrics are weakly correlated with regularizer strength and insensitive across methods — independent support for our finding that locality thresholds must be calibrated from a measured noise floor (§7), and a protocol our locality dimension should be benchmarked against. **Editing and merging methods.** Locate-and-edit methods (the ROME/MEMIT lineage) target factual associations; weight-space merging methods (Task Arithmetic, TIES, DARE) combine task vectors. We are agnostic to the patch *mechanism* — we evaluate LoRA adapters here — and orthogonal to the merging *method* — we study the naive sum as the baseline an install policy must beat. The exposure-bias gap underlying §5 is classical. Prior work documents that editing can harm general abilities and that routing among parameter-efficient experts trades off against merging; what is new here is the per-dimension certificate that turns those observations into an accept/refuse contract. --- ## 3. The Certificate Framework ### 3.1 Capability selection gate A capability is an eligible repair target only if it is **measurable** (a concrete automatable metric exists), **repeatable** (same input → same score under a fixed seed and greedy decoding), **isolated** (a locality neighborhood of behaviors that should not move can be declared), and **benchmarkable** (a held-out eval slice exists, disjoint from training). Targets failing any criterion — "reasoning," "helpfulness" — are rejected before any patch is trained. This gate prevents benchmark sprawl and is the precondition under which the locality dimension is even meaningful. ### 3.2 Evaluation protocol (non-negotiable) All capability scoring uses **greedy free generation**. Teacher forcing is permitted only to *demonstrate* its inflation (§5) and never appears in a certificate path. SQL correctness uses **set-semantics execution accuracy** (a prediction is correct iff it returns the same result set as the gold query, order-insensitive, duplicate-insensitive — the Spider convention). Every reported metric carries its sample size and a bootstrap 95% confidence interval. The base model is pinned by commit hash (`Qwen/Qwen2.5-1.5B-Instruct @ 989aa7980e…`); a certificate evaluated against a different revision is invalid by construction. ### 3.3 Dimensions and the verdict function A certificate reports four dimensions, **each adjudicated independently — they are never averaged**: - **Efficacy.** The change in target-capability execution accuracy, patched minus base, with a paired bootstrap CI. PASS iff the point estimate ≥ τ_eff (default 0.10) **and** the CI lower bound > 0. - **Locality.** The maximum absolute drift across a *declared-before-running* neighborhood of behaviors that should not move. PASS iff max drift ≤ τ_loc (calibrated; §7). Two-sided: a patch is not permitted to *move* a neighbor, in either direction, because uncontrolled side effects — even beneficial ones — signal entanglement that breaks composition. - **Regression.** Per-suite health: a perplexity ratio (≤ 1.03) and held-out general-QA drop (≥ −0.02). Perplexity is teacher-forced by nature but is a health signal, not a capability score, and is sanctioned for regression only. - **Patch cost.** Trainable parameters, delta L2 norm, storage bytes — recorded, not adjudicated. Boundary semantics are explicit and consistent: a metric exactly at its threshold PASSES, with a float tolerance so that a quantized count landing on the limit is not rejected by representation error. (This was fixed *forward-only* after we observed a boundary case; §8.4.) ### 3.4 Composition preservation Given two certified patches X and Y and an install operator ⊕, the composition test re-runs each patch's certificate measurements on the composed model X ⊕ Y. The patch is *composition-preserved under ⊕* if no dimension that held individually fails after co-installation. The baseline operator is naive merge (the sum of weight deltas); §11 evaluates alternatives. --- ## 4. Experimental Setup **Model and data.** Qwen2.5-1.5B-Instruct, SHA-pinned, in bf16 (the §5 measuring-stick law also uses the 7B variant). Spider (text-to-SQL): we render each database schema into the prompt and execute predictions against the provided SQLite databases; the second domain (§12) uses the Berkeley Function-Calling Leaderboard. We iterate on 1.5B by design and reserve larger-model runs for recorded results. Capability slices — joins, aggregation, schema-linking — are derived by classifying gold queries; locality also includes non-SQL probes (an MMLU subset; an instruction-following probe) and regression suites (WikiText perplexity; held-out MMLU). Patches are LoRA adapters. **Sampling.** Eval slices are seeded *random* subsamples, not prefixes. We discovered mid-project that Spider's dev file is ordered by database, so a first-*n* prefix is a biased database mix (base join accuracy was 0.24 on the first 100 examples versus 0.60 on items 101–150); all post-discovery runs use seeded random sampling, and we flag the one re-baselining boundary this creates. Within-run comparisons (patched vs. base, alone vs. merged on identical examples) are unaffected. **Reproducibility and discipline.** Greedy decoding makes every measurement deterministic given the seed; bootstrap CIs use a fixed seed. Certificate runs are resumable via per-pass checkpoints. Each experiment is one runnable entrypoint writing a schema-validated JSON artifact; thresholds and success criteria are pre-registered before the run. --- ## 5. The Measuring Stick: Free Generation vs. Teacher Forcing **Hypothesis (H1).** Teacher-forced token agreement overstates the capability a user obtains under free generation; therefore certificates must score with free generation. **Design.** On a set of NL→SQL probes, we score the same model two ways: (a) free-generation execution accuracy (the model writes the whole query unaided; correct iff it executes to the gold result set), and (b) teacher-forced per-token agreement (the gold query is fed and we measure argmax agreement). We compare the two with a paired bootstrap. To test whether the effect is an artifact of one model or one difficulty level, we run a 2×2 — an easy probe set (n = 40) and a harder slice (n = 24; correlated subqueries, self-joins, multi-hop joins, HAVING over joined tables) — on both Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct (the 7B in 4-bit; quantization can only lower its quality, so it is a conservative setting for an inflation claim). **Result — the baseline operating point.** On the easy set at 1.5B, teacher-forced agreement is 0.948 (95% CI [0.936, 0.961]) while free-generation execution accuracy is 0.85 ([0.725, 0.950]); the paired gap is **0.098, 95% CI [0.002, 0.213]** under set semantics (0.148, [0.034, 0.268] under stricter multiset scoring), n = 40. The gap is positive at the pre-registered significance level. **Result — the 2×2, reported honestly.** The gap is *not* constant. At 7B the easy set saturates free generation (0.975), and the gap is consistent with zero (−0.018, [−0.053, +0.039]); on the hard slice the 1.5B gap grows to **+0.352 ([+0.157, +0.544])**, while the 7B hard gap is positive but not significant at n = 24 (+0.052, [−0.058, +0.186]). We pre-registered "the gap reappears at 7B (CI excludes 0)" and that criterion was **not met**; we report it as such rather than tuning the slice until it was. Crucially, the gap did not vanish at 7B because teacher forcing became honest — teacher agreement barely moved across all four cells (0.93–0.96) — but because free generation ran out of room to fail. **Result — the inflation law.** That observation has a precise form. Conditioning each run on its per-probe difficulty tags yields ten operating points spanning free-generation *headroom* (defined as 1 − free-generation accuracy) from 0.00 to 0.43. The teacher-forcing gap is a near-deterministic linear function of headroom, **gap = −0.045 + 0.931 · headroom**, with weighted R² = 0.988, Pearson r = 0.993, and a seeded-permutation p < 10⁻⁴. The same line fits *both* model sizes independently (1.5B r = 0.992, 7B r = 0.994). The slope near 1 means almost the entire shortfall between free generation and ceiling re-appears as teacher-forcing overstatement; the intercept near 0 means the two agree only at ceiling. The 7B non-significance is thus not a failure to replicate but the **low-headroom end of the same law**: teacher forcing misleads in proportion to how fallible the model is on the task. ![The teacher-forcing inflation gap is a near-linear function of free-generation headroom, on one line across a 5× model-size range. Each point is a (model × difficulty) operating cell; marker size ∝ cell n.](/papers/figures/headroom_curve.png) **Audit.** A point estimate is not a mechanism, so we re-executed every failing free generation. The free-generation failures are *genuine model errors*: schema hallucinations (inventing a table that does not exist), aggregate misuse (an aggregate without a GROUP BY), a spurious LIMIT that drops valid rows, and convoluted-logic queries returning the wrong set. These are precisely the errors that *cannot occur* under teacher forcing, because feeding the gold token at every step prevents the model from ever emitting the spurious clause. Teacher forcing does not measure a slightly easier version of the task; it measures a different task in which derailment is impossible. **Implication.** The certificate's efficacy and locality dimensions use free generation only — and the inflation law says exactly when this matters most: in the **fallible regime**, which is the regime where a capability is worth repairing. Where a model has effectively solved a slice (headroom ≈ 0) the two metrics agree, but there one would not patch at all. Teacher forcing is least trustworthy precisely where certification is most needed. The honest measuring stick is the foundation everything else rests on. --- ## 6. A Leaky Patch, Caught and Explained **Setup.** We train a join-repair LoRA (rank 16) on join-requiring Spider queries. Training loss falls smoothly from 0.55 to 0.165 — by the signal most practitioners use, an unambiguous success. The base model's join execution accuracy is 0.380. **Certificate verdict: FAIL on three dimensions.** Efficacy gain is +0.027 with a CI that includes zero (not significant). Locality drift is 0.18 — far above any reasonable threshold. Regression's general-QA suite drops 0.035. A patch every conventional signal calls a success is refused by the certificate. **Leak audit.** We logged each probe's generated SQL and re-executed it. Of the patch's broken non-join queries, **72 of 73 are the same failure: the injection of an unwanted JOIN**. The patch learned Spider's canonical `FROM a AS T1 JOIN b AS T2 ON …` template and now applies it indiscriminately — for example, rewriting `SELECT PetID, weight FROM Pets WHERE pet_age > 1` as a gratuitous join against an unrelated table. Aggregation queries broke 35 and were fixed 8; schema- linking broke 38 and fixed 11; the join target itself moved by only +4 net. The patch did not learn "join planning"; it learned "emit joins," and the certificate's locality dimension is what exposes the difference. This audit table is the paper's central qualitative figure: the certificate's value is not the verdict alone but that the verdict is *explainable*. **Iterations and the non-isolation finding.** Mixing non-join SQL into training (a locality anchor) fixes the in-domain leak — aggregation and schema-linking drift turn positive — but flattens efficacy (−0.013); a join-weighted mix at rank 32 recovers a *significant* efficacy gain (+0.080, CI [0.007, 0.153]) but still misses the 0.10 bar and still moves the SQL neighborhood. Across the family, every patch that gained efficacy moved the neighborhood, and every patch that held the neighborhood gained nothing. The conclusion is mechanistic: a low-rank update is not a surgical edit; it reshapes the whole text-to-SQL behavior at once. LoRA produces *significant but non-isolated* repair. --- ## 7. Calibrating the Certificate A certificate is only as trustworthy as its thresholds. The locality limit had been set to 0.02 by fiat, and every measured patch landed between 0.06 and 0.44; worse, a brittle 8-item instruction probe moved by 0.125 on a *single* flip, repeatedly deciding verdicts on noise. We therefore calibrate the locality threshold from a measured noise floor, under a rule **fixed before any data was collected**. **The benign set.** We measure how much each locality probe moves under perturbations that carry *no capability intent*: (Tier 1) random LoRA-shaped deltas at controlled norms — {0.25, 0.5, 1.0, 1.41}× of a reference patch's delta norm, three seeds each, no training; and (Tier 2) seed-variant retrains of the same recipe, whose pairwise disagreement is the run-to-run variation of an "equally good" patch. **The pre-registered rule.** For each probe p, threshold_p = clamp(mean(benign_p) + 2·sd(benign_p), floor 0.02, cap 0.10); a probe whose raw value exceeds the cap is declared *unreliable* and must be replaced rather than used with a vacuous threshold. The efficacy bar is explicitly **not** calibrated — it is a product judgment, not a noise property, and lowering it after repeated failures would be the textbook Goodhart we are trying to prevent. **Findings.** Two are load-bearing. First, **the destruction is structured, not a matter of magnitude**: a random delta at the *merged* norm (the 1.41× tier) is harmless on the SQL slices, even though a real merge at that same magnitude is catastrophic (§9). The damage lives in specific learned directions, not in the size of the perturbation. Second, the apparent *seed lottery* in side effects is partly a small-sample artifact: at adequate resolution, the instruction-following cost of SQL training is *systematic* (≈ −0.07), not a coin flip. A corollary makes resolution a design constraint: a threshold of 0.02 requires a probe of at least 51 items, because one flip out of 50 already exceeds it; our 48-item probe was coarser than its own threshold, and we enlarge it to a 200-item seeded sample (one flip = 0.005) before trusting any further locality verdict. **Consequence.** With calibrated thresholds (and a coarse-grained capability — text-to-SQL as a whole, with non-SQL locality), the efficacy dimension passes for the first time in the project (+0.21–0.23 across seed variants). The certificate is *satisfiable*; the remaining failures are real. --- ## 8. The Patch-Mechanism Wall ### 8.1 The boundary With a calibrated, resolution-correct, deliberately lenient coarse certificate, three independently trained aggregation patches all clear efficacy by a wide margin but all fail locality on a consistent instruction-following regression of ≈ −0.07. The finer probe that could have rescued a borderline candidate instead *revealed* that the borderline was noise and the harm is real and reproducible. The bottleneck is not the certificate — calibrated, resolution-correct, robust — but the **patch mechanism**: LoRA cannot, at this scale, produce a capability gain isolated enough to pass. ### 8.2 The certifiable corner We then probe the unexplored gentle end of the efficacy–locality frontier (rank 4–8, one epoch, low learning rate). A rank-8 gentle patch reaches the certifiable corner: efficacy +0.20, locality drift collapsed to exactly 0.02, regression clean except for general-QA at exactly the −0.02 boundary, where it fails by a floating-point epsilon (a count of 4/200). It is one recipe tweak from a clean pass. A rank-4 patch is too gentle and loses efficacy. The lesson is the inverse of §8.1's pessimism: a *certified* LoRA patch is not impossible, it is marginal — the corner exists. ### 8.3 An integrity decision, recorded The rank-8 near-miss failed only by a float epsilon, on a real implementation bug (a regression of exactly the allowed amount should pass; strict `<` plus float representation rejected it). We chose to honor the pre-registered "this is the final attempt for these candidates" commitment: both gentle candidates stand as recorded FAILs, no certified patch is claimed, and the bug is fixed *forward only* — with a test asserting the committed near-miss artifact keeps its recorded FAIL. We report this because the discipline is the point: a framework whose verdicts can be quietly massaged at the boundary certifies nothing. --- ## 9. Composition Breaks Certificates **Hypothesis (H2).** Co-installing two individually certified patches degrades a dimension that held for each alone. **Design.** Two pure single-capability patches — A on joins, B on aggregation — each strong in isolation. We naively merge them (sum of deltas) and re-certify each capability on the merge. We use *pure* patches deliberately: H2 is about whether *independently* trained patches interfere, so anchoring one on the other's capability would pre-entangle them. **Result.** Alone, B is the strongest patch in the entire project: aggregation 0.60 → 0.85, a *significant* +0.25. Merged, **both capabilities collapse below the base model**: joins fall from a base of 0.24 to 0.07 (significant, CI excluding zero), and aggregation falls from 0.85 to 0.53, below the base of 0.60. The merge does not merely fail to preserve B's certificate; it annihilates B's gain *and* damages the base capability. A certificate is therefore **not composition-preserved** under naive merge — the second half of the central claim, demonstrated. ![Each capability is strong as a standalone patch (green) but, under naive merge (red), both fall below the unpatched base (dotted) — the certificate is not composition-preserved.](/papers/figures/composition_below_base.png) **Mechanism preview.** The damage is destructive interference: summing two broad low-rank updates, each of which reshapes the whole SQL behavior (§6), produces a combined update that is worse than either and worse than nothing. This is consistent with §7's magnitude-refutation: a random delta at the merged norm is inert, so it is the *structure* of the two learned directions, not their combined size, that destroys. --- ## 10. Predicting Conflict From the Patch Weights **Hypothesis (H3).** The composition conflict is predictable, before merging, from features computed from the two patches' weights alone — with no co-certification. **A reversed prediction, recorded.** We initially predicted that *high* subspace overlap would cause conflict (two patches writing to the same place collide). A first measurement refuted this: the catastrophic A–B pair of §9 has *low* overlap (principal-angle 0.174 versus a chance baseline of ≈ 0.073 for random rank-32 subspaces in this architecture) and near-zero directional alignment, yet its merge is catastrophic. We record the refutation and revise: the live hypothesis becomes that conflict is governed by norm structure and interference, with overlap entering with the *opposite* sign to intuition. **Features (offline, from weights).** Per layer, the LoRA delta is ΔW = BA; we compute the principal-angle subspace overlap and cosine alignment between the two patches' deltas, the Frobenius norms ‖ΔX‖, ‖ΔY‖, their sum, and an **interference ratio** ‖ΔX + ΔY‖ / (‖ΔX‖ + ‖ΔY‖) — below one when the deltas cancel (destructive), near one when additive. No GPU and no forward pass are needed. **Targets (cheap).** For nine declared pairs spanning cross-capability and same-capability, seed-variant and rank-variant combinations, we measure the worst-case efficacy degradation of the naive merge with efficacy-only passes against a fixed seeded sample. **Results.** Univariate Spearman correlations of the features with signed degradation are strong: subspace overlap **+0.85** (confirming, with the reversed sign, that *more overlap is safer*), combined norm **−0.82**, interference ratio **+0.77** (more cancellation, more harm). Two sharper findings emerge. First, **"same capability is safe" is false**: two full-rank aggregation patches collapse each other (−0.32, −0.25) just as cross-capability pairs do (−0.26 to −0.33). Second, the governing variable is **norm dominance**: the only safe merges pair a strong patch with a tiny one (−0.04, +0.01), where the merge is ≈ the dominant patch and the subordinate is harmlessly swamped; whenever the two norms are comparable — two rank-32 patches, or two gentle ones — both collapse. Conflict is destructive interference between deltas of *comparable magnitude*; a swamping patch survives. **Verdict: inconclusive, by our own criterion.** A pre-registered leave-one-pair-out linear predictor on the full feature set overfit badly (six features, nine points: cross-validated Spearman 0.017), while overlap alone reached 0.583 — just under the pre-registered 0.6 bar. We therefore report H3 as **directionally strong but formally inconclusive**, and we own the pre-registration flaw: six features for nine pairs is over-parameterized, and the missing feature the mechanism points to is an explicit norm-dominance term (min/max ‖Δ‖). A deployable predictor needs more pairs and that feature; we leave it to a follow-on. The directional signal is nonetheless immediately actionable: an install policy can refuse a merge of comparable-magnitude, low-overlap, high-cancellation patches without ever paying for co-certification. --- ## 11. Routing Beats Merging **Hypotheses.** H6: routing (keep the patches separate; serve each capability with its own patch) beats naive merge. H5: a weight-based policy that *chooses* between merging and routing beats naive merge. **Design (offline).** From the §10 grid we compute, per pair, each operator's worst-case served-capability efficacy. *Route* serves each capability with its dedicated patch run alone — by construction free of interference. *Naive merge* uses the measured merged efficacy. A weight-based policy merges only when one patch dominates (min/max norm < 0.5, an a-priori threshold declared before computing, not fitted to the nine outcomes) and routes otherwise. Criteria are pre-registered. **Results.** **H6 is supported decisively: routing beats naive merge on 8 of 9 pairs, mean margin +0.196.** Naive merge sends both capabilities below base; routing recovers ≈ 0.2 of execution accuracy by simply not combining the weights. The single non-win is a strong-plus-tiny pair where merge and route tie. **H5 is *not* supported — and the reason is the finding.** The weight-based policy is never worse than naive merge (the safety property holds), but it never *beats* routing, because **naive merge is strictly dominated**: route ≥ merge on every pair. There is no regime in which merging wins, so there is no niche for a selective-merge policy. The optimal install policy collapses to a one-liner: *always route, never merge.* The weight-based policy earns its keep only under a hard deployment constraint that forbids keeping multiple adapters, where it correctly avoids the catastrophic comparable-magnitude merges. ![Routing strictly dominates naive merge: per pair, the worst-case served-capability efficacy under merge (red) vs route (green). Route ≥ merge on every pair.](/papers/figures/route_vs_merge.png) This closes the composition arc. Naive merge is catastrophic (§9); the catastrophe is readable from the weights (§10); and the install operation that fixes it is routing, not a cleverer merge (§11). --- ## 12. A Second Domain: Tool-Call Certification Everything above is text-to-SQL. To test whether the framework is SQL-shaped or a general instrument, we certified a second, unrelated capability — **tool/function-call generation** — on a recognized external benchmark, the Berkeley Function-Calling Leaderboard (BFCL, v4 `simple` Python category, pinned by commit). The capability passes the same selection gate: the metric is a faithful subset of BFCL's AST checker (exact function name, no hallucinated arguments, each argument value matching one of the benchmark's accepted values, optional arguments omittable), scoring is greedy free generation, and the eval is a held-out split disjoint from the training pool by construction. **The base model is already strong, and its failure is isolated.** On 200 held-out probes the base model scores 0.81; function *selection* is essentially perfect (198/200 right function, zero wrong-function errors), and all of the loss is in argument *values*. This is an ideal patch target: measurable, repeatable, and isolated to one sub-behavior. **Two patch attempts, both refused — and the certificate explains why.** A first LoRA, trained on minimal canonical-argument targets, *reduced* held-out accuracy to 0.74 (paired gap −0.07): the targets systematically omitted optional arguments, so they were shorter than the model's natural calls, and the adapter learned an argument-*dropping* bias that it over-applied to required arguments too. This is a sharper version of the §6 lesson — here the patch fails on *its own objective*, and only the free-generation paired measurement reveals it. A second attempt that fixed exactly that one variable (fuller targets including optional arguments) recovered most of the loss but landed at break-even (0.795–0.81, gap not distinguishable from zero). Naive supervised fine-tuning on benchmark answers fixes roughly as many held-out probes as it breaks. The efficacy dimension **fails** for both: the patch-mechanism wall of §8 recurs in a second domain. **The certificate also catches a leak the patch was not suspected of.** The full certificate for the break-even patch fails on a second dimension as well: locality. Its declared neighborhood includes a cross-capability probe (Spider join execution accuracy), MMLU, and instruction-following. Join accuracy moves +0.04 and MMLU −0.005 (both within noise), but **instruction-following drops 0.094** — a real collateral regression in a behavior the tool-call patch has nothing to do with. This failure is robust to the (provisional, pending-calibration) locality threshold: 0.094 exceeds any plausible calibrated floor. So the patch is simultaneously **useless on its target and harmful off it**, and the per-dimension certificate states both facts where training loss and a single accuracy number would have shown a clean-looking result. **A repeatable cross-capability effect.** The instruction-following regression is not a one-off: SQL capability training degraded instruction-following by ≈0.07 (§7), and tool-call training degrades it by 0.094. Two unrelated capability patches, the same collateral victim — evidence that capability SFT generically erodes general instruction-following, exactly the kind of effect a locality dimension is meant to surface and an averaged score would hide. The framework transfers: a different capability, a different external benchmark, the same machinery, and the certificate again refuses a patch every conventional signal would accept. ![Second domain (tool-call / BFCL). Left: naive patches do not beat the base (v1 reduces accuracy; v2 is break-even). Right: the break-even patch nonetheless leaks, dropping instruction-following by 0.094 past the locality threshold — the certificate FAILs on efficacy *and* locality.](/papers/figures/toolcall_cert.png) --- ## 13. Discussion **The certificate is the validated contribution; the registry is the open question.** Across the study, the certificate did its job at every step — it refused a loss-looks-great patch, it caught composition collapse, it was made satisfiable by calibration, and it held the line at the boundary. What remains genuinely open is whether *any* cheap patch mechanism can produce a *certified* patch: LoRA reaches the certifiable corner but lands on its edge. This re-frames "a registry of certified patches" from a deliverable we are failing to hit into the precise research question our validated instrument now lets us ask — and points to isolation-by-construction mechanisms such as null-space- constrained editing, applied at capability rather than fact granularity, as the natural next patch family to put through the certificate. **Composition has a structural answer and a predictive one.** The structural answer — route, do not merge — is simple and strong. The predictive answer — conflict is readable from norms, overlap, and interference — lets an install policy refuse or route *before* paying for co-certification, which is the expensive step. Together they are the composition-preservation product the framework set out to build. **Honesty as a load-bearing component.** Pre-registration and an append-only journal let us report a near-miss as a FAIL (§8.3), a reversed prediction as a refutation (§10), and a probe-resolution defect as a forward-only fix (§7) — without spending credibility. A certification framework whose verdicts can be quietly tuned certifies nothing; the discipline is not adjacent to the contribution, it is part of it. --- ## 14. Limitations - **Scale coverage is partial.** The measuring-stick law (§5) is established across 1.5B and 7B, but the certification, calibration, and composition results (§6–§11) are on Qwen2.5-1.5B-Instruct only. The non-isolation of LoRA and the magnitude of composition collapse may differ at 7B+; those runs are in progress, not yet recorded. - **Two domains; composition on one.** Certification now covers two capabilities — text-to-SQL on Spider and tool-calls on BFCL (§12) — but the composition results (§9–§11) are SQL-only, and both domains are at the 1.5B scale. A third domain and cross-domain composition (SQL × tool-call) would further test generality; the BIRD generalization slice the certificate schema was designed around is also deferred. - **Small-n conflict analysis.** The §10 predictor rests on nine pairs sharing a single joins patch; it is directional, not a deployed model, and a clean fit could partly be same-vs-cross-capability classification in disguise — which we flag per pair. - **Oracle routing.** §11 assumes a correct query→capability router; a learned router is a separate component we specify but do not build. - **No certified PASS yet.** The strongest patch is a boundary near-miss. We claim the certifiable corner exists, not that we have populated the registry. - **Concurrency.** §10 overlaps with concurrent work; we differentiate but cannot claim priority on merge-predictability per se. --- ## 15. Conclusion Capability patches cannot be trusted on the strength of training loss or a target metric: they leak, and they do not survive composition. We make the **certificate** — a calibrated, per-dimension, free-generation behavioral statement with composition preservation — the unit of trust. On text-to-SQL we show the certificate catches a patch every conventional signal endorses, that its thresholds must and can be calibrated from a measured noise floor, that naive composition drops two strong patches below base, that the resulting conflict is predictable from the patch weights, and that the safe install operation is routing rather than merging. The patch is an artifact; the certificate, and the proof that it survives composition, is the product. --- ## Appendix A — Reproducibility Base model pinned by commit SHA; greedy decoding; bootstrap CIs at fixed seed; seeded random eval sampling (not prefixes). Each experiment is a single runnable entrypoint writing a schema-validated JSON artifact, with resumable per-pass checkpointing. The complete pre-registration-and-results journal, all certificate artifacts, the calibration data, the conflict grid, and the install-policy evaluation are kept alongside the code and available on request. Unit tests cover the pure-logic core (scoring, verdict boundary semantics, calibration rule, predictor, policy). ## Appendix B — Pre-registration log (selected) - Locality calibration rule (clamp(mean + 2sd, 0.02, 0.10); unreliable above cap) — declared before calibration data. - text-to-SQL@2.0 re-cert protocol (fresh eval seed, any-PASS-counts, last amendment for these candidates) — declared before the run. - Probe-resolution fix (instruction probe n→200 seeded; thresholds unchanged) — declared as a measurement fix, final amendment. - H3 predictor verdict (LOO |ρ| ≥ 0.6 and overlap+norm beats overlap-alone; refuted if < 0.3) and H5/H6 criteria (a-priori dominance threshold 0.5) — declared before computing. ## Appendix C — Calibrated thresholds (cert-spec, artifact-level) Locality, from the benign placebo floor: joins ≈ 0.030, aggregation ≈ 0.047, schema-linking ≈ 0.040, MMLU ≈ 0.020, instruction-following ≈ 0.020 (with the enlarged n=200 probe). Efficacy bar 0.10 (with CI excluding zero), uncalibrated by design. Regression: perplexity ratio ≤ 1.03; general-QA drop ≤ 0.02. Boundary values pass, float-tolerant. --- ## References A full bibliography will accompany a later revision. Works referenced in the text include: *The Mirage of Model Editing*; *Are We Evaluating Edit Locality Properly?*; *Provably Safe Model Updates*; *Demystifying Mergeability*; the ROME / MEMIT model-editing lineage; Task Arithmetic, TIES, and DARE weight-merging; LoRA; AlphaEdit (null-space-constrained editing); and the Qwen2.5, Spider, BFCL, MMLU, and WikiText models and datasets. --- ## Research notes — Perception URL: https://perception.club/notes Notes # Research notes Short-form writing on the techniques we work on and what we learn along the way. - 2026-09-17 ### Point it at a warehouse, get an analyst trained on the answer Root-cause analysis has no training signal: a warehouse records what happened, never why. So we generate a simulated twin of the warehouse where the answer is known exactly, train an investigator there, and send it back to the real data read-only. One measured run: 27 minutes from an unseen database to a trained investigator, 55k tokens per investigation after that. - 2026-09-16 ### A swarm of agents, trained on reward: better answers, five times the cost We replaced the single investigating agent with a team - a lead that plans and assigns, three specialists working in parallel - on the same 20 held-out investigations. Training the team gave the clearest improvement we have measured (60% to 80% fully correct, p = 0.023), but the trained team still did not beat one trained agent on the scoring rubric: more accurate, at 4.9 times the tokens. - 2026-09-15 ### Pilot: an agent that learned from reward investigated as well, at 40% of the cost A pilot in a simulated company: an investigation agent that learned a playbook from environment reward matched the best untrained agent (70% fully correct) with 60% fewer tokens, was best on cause types it never trained on, and learned one rule that went too far. Too small to be significant; the full test plan is below, and results will follow. - 2026-09-15 ### Reward alone taught a frozen model the house rules A frozen gpt-5.4 learned undocumented accounts-payable conventions from environment reward, going from 0% to 98% fully correct on held-out invoices without a single weight update. The first learner reached 26%; why it stalled was the most useful finding. - 2026-06-16 ### You can’t trust a patch without a certificate The concluded study: calibrating the certificate so its verdicts mean something, why low-rank patches land on the edge of certifiability, route-don’t-merge as the settled install policy, and the framework holding up across a 5× model-size range and a second, unrelated domain. - 2026-06-11 ### Certified capability repair: what we’ve learned so far First results from our repair program: honest capability measurement, a certificate that correctly refused a patch that looked great by training loss, and proof that naively composing two good patches destroys both. --- ## Point it at a warehouse, get an analyst trained on the answer — Perception URL: https://perception.club/notes/investigation-studio Note · 2026-09-17 # Point it at a warehouse, get an analyst trained on the answer Every company asks why a number fell, and no company records the answer. That is the whole problem with training an agent to investigate: your warehouse knows what happened, never why. Last Tuesday’s drop has no label, and without a label there is no reward. This note describes what we built to get round that, and walks through one execution of it end to end, with the numbers from that run rather than from a description of it. ## The problem “Revenue fell 14% last week — why?” is one of the most common questions in any business and one of the hardest to answer well. An analyst decides where to look first, reads what comes back, forms competing explanations, tests them, drops the wrong ones, and stops when the evidence is enough. There is no single correct path, and a wrong first move wastes the whole investigation. A capable model with database access can already do this passably. The question we care about is whether it can learn to do it better from its own outcomes — and that runs into the wall above. The usual workarounds make it worse rather than better. An LLM judge scores confident writing rather than correct answers. Human labels cost more than the analysis they replace. Historical incidents are few, inconsistent, and written up after the fact by whoever was wrong first. ## The approach We generate a twin of the warehouse: the same table and column names, the same dimension vocabularies, roughly the same scale, and entirely synthetic rows. Into that twin we inject causes. Because the world is generated deterministically, it can be rebuilt with one cause removed and everything else identical — and the change in the metric is that cause’s exact contribution. That is the reward, and it is arithmetic rather than opinion. Figure 1. Only the first and last steps touch real data, and both are read-only. Training happens entirely in the simulation. What crosses from the simulation to the real data is not a model — the weights never change — but a policy state: a system prompt and playbooks of learned lessons, in plain English, which a person can read before it is pointed at production. ## Step 1 and 2: connect, and work out the use case Two things are supplied by a person: a connection string, and which number matters. llm-rle studio init duckdb:///warehouse.duckdb --list-kpis llm-rle studio init duckdb:///warehouse.duckdb --kpi 0 \ --kpi-name "weekly paid invoice revenue" --out specs/revenue.yamlProfiling reads every table, column, value range and vocabulary — never rows. Inference then works out the fact table, the metric and its date column, the dimensions worth slicing by, the entity table behind the fact rows, and which tables can serve as evidence for which kinds of failure. Figure 2. Run against a SaaS warehouse written independently of the framework, in a different domain from anything it had seen. All seven role assignments correct, from the schema alone. Two decisions carry most of the weight here. Inference is rules, not a model: a model that invents a plausible column name produces a world an agent then trains in, so a model is used only to write business-language descriptions, and every table and column named is checked against the schema before anything is generated. And the result is a spec file that a person reviews — which metric, which slices, and what the generator believes can go wrong in this business. Failure mechanisms are gated on what the schema can express. A warehouse with no currency column never gets currency-translation faults; one with no load-audit table never gets pipeline gaps. In this run, seven of the ten mechanisms in the catalogue were available and the other three are named in the spec as out of scope, so the spec is honest about what it does not cover. ## Step 3: generate a world that can be rerun Figure 3. Each cause's share, measured by rebuilding the same week without it. The decoys measuring exactly zero is the check that the simulation is sound. Tasks are generated across the splits that separate a lucky agent from a capable one: familiar cause types, two causes at once, cause types deliberately held out of training, and cases with a decoy planted to mislead. Each generated investigation reads like the real thing, and carries an answer the agent never sees: test_recovery-0 — weekly paid invoice revenue for 2026-09-07 to 2026-09-13 was $270,427, 28.8% below the average of the previous 4 periods ($379,813). Investigate the data, identify the root causes of the decline, estimate each cause’s share, and submit your conclusion. Truth, never shown: a processing failure scoped to currency=AUD worth 1.20 of the decline, plus two decoys worth exactly nothing. Generation tunes itself to the warehouse, because the same “85% of this slice fails” is a rounding error in one business and an outage in another. Magnitudes are tuned until the metric falls believably, a cause whose slice is too small to matter is widened, and a week whose own variation would credit a cause with more than the whole decline is re-rolled. Fourteen tasks for this use case took 23 seconds and no model calls. ## Step 4: train, and read what was learned llm-rle studio train specs/revenue.yaml --policy single llm-rle studio train specs/revenue.yaml --policy swarmBoth arms train with the model’s weights frozen. What improves is a playbook, learned by contrasting a high-reward investigation against a low-reward one on the same case and writing down what separated them. The single agent learned eight lessons; the first, verbatim: Start with broad decomposition: run describe_schema, then kpi_breakdown on a few high-yield dimensions (segment, region, payment_method, currency) and identify the single slice contributing most of the KPI decline. The swarm is a lead plus three specialists whose briefs are written from this warehouse’s own tables — one owns breakdowns, one owns the accounts behind the metric, one owns whether the value was lost or merely not recorded. Each keeps its own playbook plus a shared one. Its measurement specialist learned: For a KPI defined on paid invoices, start with one compact invoices check comparing current vs baseline by status, count(*), and sum(amount_usd). On eight unseen tasks in the twin, the single agent scored 0.50 reward and 38% fully correct; the swarm 0.29 and 12%, at nearly double the tokens. With six training tasks and one seed that is a direction rather than a verdict — but it does show the generated environment discriminates instead of scoring everything alike. ## Step 5: investigate the real warehouse llm-rle studio investigate specs/revenue.yaml --db duckdb:///warehouse.duckdb \ --policy single --policy-state runs/…/policy/final.json --out report.mdThe agent is told what the metric actually did in the most recent complete period and investigates with the same tools it trained with. There is no ground truth here and the framework does not pretend otherwise: nothing is scored. The output is a conclusion — each cause with a scope, a share and its evidence — followed by every query it ran, in order, for a person to check. From the generated report, after 28 queries and 48 seconds: The 7.0% KPI decline was driven almost entirely by a processing problem in EMEA card payments. After the 2026-09-06 billing deployment that migrated EU card processing to a new provider, the count of paid EMEA card invoices fell sharply, while invoice values did not — pointing to transactions failing to complete rather than weaker demand or lower pricing. That conclusion is correct. The demo warehouse was built with exactly that fault, and with a small-account churn in the same week as a decoy, which the agent examined and rejected. It was told neither. ## One execution, end to end Figure 4. Every number measured on the run. Training is paid once; investigations afterwards cost 55k tokens each. Twenty-seven minutes from an unseen database to a trained investigator, of which four and a half seconds were spent reading the schema and twenty-three generating the training world. Both of those steps make no model calls at all. ## The case that separates them A single-cause week turned out to be a poor test: all three arms found it, and it separated nothing. So the demo warehouse also carries a week with three simultaneous causes from three different families — an aborted billing load, enterprise accounts churning, a misconfigured promotion — alongside three decoys that move and cost nothing. Because it is simulated end to end, each share is exact and the arms can be scored on data that looks real. Figure 5. Each arm misses a different cause, and none of them solves it — which is what a benchmark is for. Two honest notes travel with that table. With one investigation per arm it is not a ranking. And the promotion was scoped to a plan tier that lives two joins from the metric, so inference never made it a dimension and no agent could have named that scope even after finding it — a real limitation of the current system, written down rather than tuned away. ## What is enforced, and what it cannot do Four properties are enforced rather than intended. The connection reads only: every statement is checked before it runs — one statement, SELECT or WITH only, no DDL or DML — and is opened read-only where the driver allows. There is no tool that writes: the agent can query and submit a conclusion, and nothing else. Your data stays put: the spec carries schema, dimension vocabularies and summary statistics, never rows, and never free text — notes, titles and descriptions stay in the warehouse. Credentials never enter a spec or a report. What it cannot do yet, in the order the limitations bite: the twin generates a star schema, so event-sourced or heavily normalised warehouses are simulated more loosely; dimension inference walks one join, so a slice defined two hops away cannot be named; one metric per spec; and training quality is bounded by whether the catalogue’s mechanisms resemble how the business actually fails, which is the judgement the spec review exists for. One more thing worth saying plainly. The first training run in a generated twin scored 0.016 mean reward with zero successes, while the same untrained agent investigated the real warehouse correctly. Every reason was in the generator or the grader, not the agent: all-or-nothing scoring of near-miss labels; a noise floor of 5% against a 4% minimum decline, so a “task” could be pure variation; a world that grew into the question; and magnitudes fixed in the catalogue rather than tuned per warehouse. Two more were caught before that — a per-process random seed that made cached tasks describe data that no longer existed, and the twin copying the real warehouse’s note text into the spec. A generated environment is a piece of software like any other, and it is wrong until it is checked. ## Where this goes Three things next, in order. Breadth: a second and third warehouse in unrelated domains, to test the inference rather than the twin — the generator is the easy half, recognising an unfamiliar schema is the hard one. Depth: more training budget on one use case, several seeds of the three-cause shape as held-out tasks, and a cost-matched swarm arm, enough to turn “0.80 against 0.61” into a result rather than an anecdote. Reach: two-hop dimensions, more mechanisms in the catalogue, and a scheduled mode that investigates the weekly number and writes the report before anyone asks. This generalises the hand-built arena behind two earlier notes — the pilot and the swarm comparison — from one environment somebody wrote by hand to one the framework writes for whatever warehouse it is pointed at. If you work on investigation or root-cause analysis over a data warehouse, we would like to hear from you. All notes --- ## A swarm of agents, trained on reward: better answers, five times the cost — Perception URL: https://perception.club/notes/swarm-of-agents-trained-on-reward Note · 2026-09-16 # A swarm of agents, trained on reward: better answers, five times the cost When an agent struggles with a hard, open-ended task, the popular answer is to use more agents: a lead that plans, specialists that work in parallel, a synthesis at the end. We had an environment where we could check that claim rather than assume it, so we did. The short version: a team of agents is not automatically better. Untrained, the team scored lowest of the five agents we have now measured in this environment. Trained on the environment’s reward — every member keeping its own playbook of lessons — it became the most accurate: 80% of held-out investigations fully correct against the single trained agent’s 70%, with the best evidence quality of any arm and not a single decoy claimed. It also spent 4.9 times the tokens to get there, which on the scoring rubric cancels the gain exactly. This continues the pilot published yesterday, and uses the same company, the same 20 held-out investigations, the same tools and the same grader. Only the thing holding the tools changed. ## Objective In the first pilot, a single agent investigated: it chose what to query, read the results, formed hypotheses and submitted a conclusion. Training it on reward made it dramatically more efficient at the same accuracy. A team is a different bet. Three specialists can examine three explanations at once, in separate contexts, so more evidence reaches the conclusion and no single context has to hold all of it. That is the argument. The counter-argument is just as plausible: three specialists produce three partial stories, and someone has to decide between them without having seen the data themselves. We wanted to answer two questions with the same measurements we already had: - Does dividing an investigation among specialists beat one agent doing all of it? - Does learning from reward help a team more or less than it helps a single agent? ## Plan The team is a lead investigator and three specialists. The lead reads the case, splits it into three lines of inquiry and assigns one to each specialist. The specialists work at the same time, each in its own context, with the same tools the single agents had, minus the ability to submit: only the lead concludes. Each reports back what it found, the numbers behind it, and what it ruled out. The lead names what is still missing, can send one follow-up round, and then submits one conclusion, which is what gets graded. Figure 1. One investigation, run by a team. The specialists never submit; the lead never queries the data. What learns. In the trained arm, each member keeps its own playbook, and there is one shared team playbook that everybody reads. Reward arrives only at the end, for the team’s single conclusion, so credit has to be assigned. We do it the same way the single-agent learner does — contrast a high-reward investigation with a low-reward one on the same case — but show that contrast one member at a time, through only that member’s own turns. Lessons about dividing work and handing over go to the shared playbook instead. The model’s weights never change; the playbooks are the whole checkpoint, and you can read them. What stayed fixed. The company, the 20 held-out investigations, the four test types (familiar cause types, two causes at once, cause types never trained on, and a decoy planted to mislead), the tools, and the grader — including its efficiency penalty, which charges for every tool call beyond fifteen. A team makes more calls than one agent, so that penalty costs it more. We left it alone: the cost of an answer is part of the answer. ## Experiments Two new arms, both on the same 20 held-out tasks: the team as prompted roles with no learning, and the same team trained on the same 12 training cases and the same budget the single trained agent got. Before either arm ran, we tried one investigation and read the whole transcript. It caught a flaw of our own making: specialist reports were being cut to a third of their length before the lead read them, and the lead said so in its own review — it complained that the reports it had been given were truncated. Reports now reach the lead whole. One flaw we found later we deliberately did not fix: see the over-claiming in the outcomes below. Both swarm arms had to run identical prompts, or the comparison between them would mean nothing. Everything was re-graded together with the three arms from the first pilot, so all five are scored by the identical verifier. ## Outcomes armfully correctrewardaccuracy onlyroot-cause F1attributiondecoy claimstool callstokens A. SQL only50%+0.77+0.860.860.760.202961k B. investigation tools70%+0.82+0.920.940.830.053694k C. one agent, trained70%+0.82+0.840.870.760.051738k D. swarm60%+0.64+0.800.860.630.0557141k E. swarm, trained80%+0.80+0.950.970.900.0052185k “Accuracy only” is the same rubric with the efficiency penalty removed, to separate how right an arm was from what it spent. Figure 2. The trained team is the best or level on every test type. Both swarm arms handle cause types never seen in training. Training the team is the clearest effect we have measured in this environment. Trained against untrained, on the same 20 investigations: +0.163 reward, 95% confidence interval [+0.039, +0.317], p = 0.023, and fully-correct investigations up from 60% to 80%. Unlike every comparison in the first pilot, this one is measured across all 20 tasks rather than 5, which is why it can clear the significance bar at all. The team did not beat one trained agent. On the rubric they are indistinguishable: 0.798 against 0.820, p = 0.75. The team is genuinely more accurate — it identified causes better (F1 0.97 against 0.87), sized them better (attribution 0.90 against 0.76), and was the only arm that never once claimed a decoy — and it paid for that with 185k tokens per investigation against 38k. Figure 3. What the extra accuracy costs. Ten points of accuracy is two investigations out of twenty. Why the untrained team was the worst arm of all. It had the same tools as arm B and scored well below it. The transcripts show a specific failure: three specialists each report findings from their own slice, and an untrained lead passes them all through instead of deciding between them. It claimed 3.4 causes per investigation against the single trained agent’s 1.4. In the worst case it split one cause into sixteen slices by category and channel, each with a tiny share, together adding to 0.17 of the decline instead of 1 — the grader gave it full marks for finding the cause and nothing for sizing it. Figure 4. Training taught the lead to decide rather than collate. The same case, before and after. Each member learned something only it could use. The lead learned to separate demand from post-order execution before naming anything; the demand analyst to put booked demand beside completed revenue, which tells “customers stopped buying” apart from “orders stopped completing”; the supply analyst to localise to a single warehouse and require three aligned signals before concluding; the pricing and data analyst to check whether the reporting pipeline itself explains the gap before blaming the business. The shared playbook holds only coordination: assign each analyst a competing explanation rather than a domain, make every handoff decision-ready, do not stop at the first large driver. Figure 5. The whole checkpoint is readable. No lesson names a region, account or date from any training case. What training cost. It fixed six investigations and broke two, and both regressions have the same signature: the trained team adds a partner-loss claim that is not a true cause, because its specialists are now told to report partner and customer activity and the lead sometimes promotes that report into a claim. Learning from reward changes behaviour in more places than the one you were aiming at, which is why the held-out set and the per-task diff matter more than the headline average. ## Conclusion If you are choosing between one agent and a team for this kind of work, the pilot says: a team is worth it only if you train it, and only if accuracy is worth about five times the cost. An untrained team of capable agents was worse than one capable agent — not because the model is weak, but because dividing work creates a decision problem that nobody in the team is responsible for solving. Training put that responsibility somewhere: the lead learned to pick, and claims per investigation fell from 3.4 to 1.65. The result we did not expect is where the benefit landed. The team’s edge shows up on the two hardest test types — misleading decoys (80% against 60%) and familiar causes it had been getting wrong (100% against 80%) — and vanishes on combined causes, where the plain SQL agent’s 60% is still the best any arm has managed and the four since have all sat at 40%. Two causes at once remains the unsolved case here, for one agent and for four. Both arms together cost 10.2M tokens and 46 minutes. Next, in order: tell the lead to consolidate — one claim per cause, at the narrowest scope the evidence supports, shares summing to about 1 — which addresses both the over-claiming and the spurious extra claim; then a cost-matched arm that caps the team at the single agent’s token budget, because “is a team better at equal cost?” is the question these numbers actually pose; then the same comparison with a cheaper model, where a team of inexpensive agents against one expensive agent is the trade that would matter in production. As with the first pilot, five tasks per test type is too few for any per-type difference to be significant, and this is one seed and one model. The full test — two models, three agents each, 100 held-out tasks — is still planned, and its results will be published whether or not they support what we found here. If you are working on agents for investigation or root-cause analysis, we would like to hear from you. All notes --- ## Pilot: an agent that learned from reward investigated as well, at 40% of the cost — Perception URL: https://perception.club/notes/pilot-learning-to-investigate Note · 2026-09-15 # Pilot: an agent that learned from reward investigated as well, at 40% of the cost “Why did revenue fall 12% last week?” is one of the most common questions in any company, and one of the hardest to answer well. An analyst has to decide where to look first, read what comes back, form competing explanations, test them, drop the wrong ones, and stop when the evidence is sufficient. We built an environment to ask whether an agent can learn to investigate like that from outcomes, and ran a pilot. The headline, with the caveat first: this is a pilot with five held-out tasks per test type, so nothing below is statistically significant. Within it, an agent that learned from reward matched the accuracy of the best untrained agent (70% of investigations fully correct) while using 60% fewer tokens and half the tool calls, and it was the best agent on cause types it had never seen in training. It also learned one rule that generalised too far. The plan for the full test, and what would count as success, is at the end; its results will be published in a follow-up note. ## Objective Root-cause investigation is a sequential decision problem rather than a generation problem. There are many possible sequences of actions, every action returns new evidence, early choices shape what is worth doing next, there is no single correct path, and a good investigator has to recover from hypotheses that turn out wrong. Success, though, is measurable: did the investigation find what actually caused the decline? A capable model with tools can already investigate. At each step it picks a reasonable next query. The question we care about is different: can an agent learn, from the outcomes of its own investigations, a better way to investigate? And is any improvement due to that learning, or just to giving it better tools? For the pilot, “learning” means the approach from our earlier note: the model’s weights stay frozen, and it learns a playbook of lessons from environment reward. Training the weights themselves on the same environment is a later step. The pilot’s job was to check that the environment and its scoring are sound, and to see which way the results point. ## Plan ### A company to investigate The Business Investigation Arena generates a company for every task: 19 tables and about 750,000 order lines across a data warehouse, operational orders, shipments, inventory, supplier receipts, CRM accounts and notes, support tickets, the web funnel, marketing spend, exchange rates, data-load logs and a company event feed. Into each company we inject one or more real causes of a revenue drop — a stock-out, a warehouse outage, a price increase, a misconfigured promotion, a competitor launch, a marketing freeze, a lost reseller, key accounts leaving, payment failures, severe weather — together with decoys: evidence that looks relevant but moved no revenue, such as a campaign ending on schedule or an analytics tag under-counting web sessions. Because the simulator reuses its random draws, the true share of the decline caused by each cause is exact: re-run the same week without that cause and measure the difference. The agent receives only the question, for example “Revenue for the APAC region in week 2026-W37 (2026-09-07 to 2026-09-13) was $8,568,041, 12.4% below the average of the previous four weeks ($9,781,946). Leadership wants to know why.”, and must submit each cause with its scope and its share of the decline. ### Four kinds of test - Familiar causes: the same kinds of cause the agent trained on, in new companies. - Combined causes: two familiar causes at once, so their effects must be separated. - Unseen cause types: causes never present in training — a currency move that changes dollar-reported revenue, a data-load failure that hides sales from reporting, and a supplier delay that cascades into stock-outs and lost accounts. - Misleading decoy: a familiar cause plus a decoy chosen to point at the wrong explanation. ### Three agents Figure 1. One environment, three agents. B and C have identical tools, so the difference between them is training alone. All three agents use the same model (gpt-5.4) and can reach exactly the same data. Agent A has the minimum a normal data agent gets: the schema, SQL, and a way to submit. Agent B adds a semantic analytics layer — breakdowns of the change by any dimension, a decomposition of a drop into volume, price, currency, fulfilment and reporting effects, entity lookups, event search and a hypothesis log. Agent C is agent B after training in the environment. Comparing B with A measures the tools; comparing C with B measures the learning. ### Scoring A verifier compares each submission with the simulator’s ground truth: whether each real cause is found with the right type and scope, how close its claimed share of the decline is, whether the agent actually examined the evidence for it, and penalties for naming decoys, inventing entities or wasting queries. An investigation is fully correct only if it finds every cause with the right scope, gets each share within 0.2 of the truth, and claims no decoy. ## Experiments Modelgpt-5.4 via Azure OpenAI, default sampling Held-out evaluation20 investigations: 5 per test type, identical for all three agents Budgetup to 25 tool-using turns per investigation Training (agent C)12 training companies with familiar causes only; 2 learning steps of 6 tasks × 2 attempts; the playbook is updated by reflecting on better and worse attempts and reading the true answer Spendabout 5.9M tokens recorded across the three agents and a live smoke test ### Reading transcripts before trusting the numbers Before comparing agents, we read their investigations against the ground truth. Three times, the scoring was wrong rather than the agents: - Shares. Agents estimated a cause at about 71% of the observed decline; ground truth said 100%, because it divided by the total impact of the injected causes instead of the decline the question asks about. Shares are now a fraction of the observed decline. - One cause, described twice. When an agent reported a single promotion error separately for online and partner orders, the second half counted as a false claim, and a 3% side note cost as much as a wrong 50% cause. Claims that identify the same cause are now combined, and false claims are weighted by the share they claim. - Ambiguous labels. The SQL agent correctly described a data-load failure in its written summary but filed it under “tracking or measurement issue” rather than “data pipeline issue”, and scored zero. That near miss now earns partial credit, and every cause label has a definition for the full test. The trained agent’s first run started under the second flaw, so we stopped it and trained again after the fix. All three agents are then scored by identical rules. It is the same discipline as our certificate work: the measuring instrument has to be checked before its verdicts mean anything. ## Outcomes Figure 2. Fully correct investigations by test type. The trained agent is best on unseen cause types and level overall. AgentFully correctMean rewardTokens per investigationTool calls A. Normal agent, SQL only50%0.76961k29 B. Normal agent, investigation tools70%0.82294k36 C. Agent B, trained in the environment70%0.82038k17 Better tools helped less than they first appeared. Agent B beat agent A by 0.05 in mean reward (95% interval −0.02 to +0.12). Before the labelling fix, B’s lead on unseen cause types looked like +0.59; afterwards it was +0.18. Much of the apparent advantage was A choosing a neighbouring label for a correct diagnosis. Training matched accuracy at 40% of the cost. Agent C was fully correct as often as agent B while using 38,000 tokens and 17 tool calls per investigation instead of 94,000 and 36. Even during training, investigations became cheaper between the two learning steps (77,000 to 54,000 input tokens per attempt) while training reward rose from 0.80 to 0.84. On unseen cause types, agent C was fully correct on all five tasks. Figure 3. Cost against accuracy. Training moved the agent left, not up. The difference is easiest to see on a single task. On a held-out investigation where a misconfigured promotion caused the drop and a campaign ending that week was the decoy, the untrained agent made 39 tool calls and blamed the campaign for a quarter of the decline. The trained agent made 17, decomposed the change early, and gave the campaign 3%. Figure 4. The same held-out task, investigated before and after training. Each block is one tool call. It also learned a rule that went too far. The trained agent’s playbook is readable, which makes both its strengths and its mistakes inspectable. Most lessons describe a sensible investigation strategy: run broad cuts, decompose the change to classify its mechanism, confirm with one independent source, and stop. One lesson, written after a competitor launch in training, told it to prefer the external event over “the downstream symptom of fewer buyers”. On two held-out tasks where key accounts had genuinely left, it found the right accounts, read CRM notes saying they were moving to a competitor, and labelled the cause a competitor action. Both untrained agents got those tasks right. They account for the whole of its shortfall on familiar causes. Figure 5. Verbatim lessons from the trained agent's playbook, including the one that over-generalised. None of this is significant yet. With five tasks per test type, the smallest p-value a paired sign-flip test can produce is about 0.06, so no difference within a test type could reach significance, and overall the trained and untrained agents are level (difference in mean reward −0.002, 95% interval −0.16 to +0.12). The pilot shows direction, not proof. ## Conclusion What the pilot shows. The environment works end to end: companies with exact ground truth, investigations that run and submit, a verifier whose mistakes we could find and fix, and an agent that improves from its own outcomes. Learning from reward made the agent markedly more efficient without losing accuracy, and its best results came on causes it had never trained on, which is where learning an investigation strategy should matter most. What it does not show. Five tasks per test type, one run, and only 24 training attempts cannot establish that learning beats a well-equipped untrained agent. Familiar causes are already solved by the untrained agents, so they cannot show a gain. The one over-general lesson is a warning: learning from a handful of tasks can also teach the wrong rule, and the pilot ran without the regression check designed to catch that. ## Next: the full test Figure 6. The full test, planned after the pilot. Results will be published in a follow-up note. The full test keeps the pilot’s design and fixes what the pilot exposed: - Two models. The three agents on gpt-5.4 and on the smaller gpt-5.4-mini, to see whether learning matters more for a weaker model. - Enough tasks to decide. 100 held-out investigations per agent, 25 per test type, identical for every agent. - More, and checked, training. At least 48 training companies over two epochs, with a regression gate that keeps a new playbook only if it does no worse on fresh training tasks. - Clear labels. Every cause label carries a definition, so a wrong label is a real error rather than an ambiguity. We fixed three hypotheses before running it: - H1. The trained agent beats the untrained agent with the same tools on combined causes, unseen cause types and misleading decoys, while losing no more than five points of accuracy on familiar causes. - H2. The trained agent needs substantially fewer tokens per investigation at equal or better accuracy. - H3. The gain from learning is larger for gpt-5.4-mini than for gpt-5.4. Every comparison will be paired on the same tasks and reported per test type with bootstrap intervals and permutation tests. We will publish the results in a follow-up note, including any hypothesis that fails. If your team spends its time explaining why a metric moved, talk to us. All notes --- ## Reward alone taught a frozen model the house rules — Perception URL: https://perception.club/notes/reward-alone-taught-a-frozen-model-the-house-rules Note · 2026-09-15 # Reward alone taught a frozen model the house rules Every organisation runs on conventions its documents never state: how a supplier’s name is written in the ledger, what an invoice number looks like, which account a purchase belongs to. A general-purpose model cannot know them, and that is usually where fine-tuning starts. We asked a narrower question first. If an environment can score the work, how much of that gap closes from reward alone, without touching the model’s weights? On a synthetic accounts-payable task, a frozen gpt-5.4 went from 0% to 98% fully correct extractions on 100 held-out invoices in six learning steps, matching a copy of the same model that was simply handed the rules. The first version of our learner only reached 26%. Why it stalled turned out to be the most useful finding of the study. ## Objective The practical question: can a model learn an organisation’s unwritten conventions from an environment that checks its output, rather than from examples it is trained to copy? Fine-tuning answers this by collecting labelled examples, training, hosting the result, and repeating the cycle whenever a rule changes. An RL environment answers it differently: let the model attempt the real task, verify what it actually produced, and reinforce what worked. Our longer-term work applies that loop to model weights. This study isolates the environment half — the task, the verifier and the evaluation discipline — and pairs it with the cheapest learner we could build: one that never updates weights at all, and instead writes what it learns into a playbook the model reads. We set three conditions for calling the result a success, the same discipline we use for certificates: - Held-out gain. The improvement has to appear on tasks the learner never saw, under free generation. - No silent regression. Every update is checked against fresh tasks before it is kept. - A fair frame. The number must sit between a no-learning control, which measures noise, and an oracle, which measures the ceiling. ## Plan ### An environment with rules nobody wrote down The task is invoice extraction. Each episode shows the model one supplier document and asks for seven fields as JSON: vendor, invoice number, date, total, currency, payment terms and expense category. The request names the fields and nothing else — deliberately under-specified, the way real business requests are. The reference records follow house conventions the model is never told: - Vendor: upper case, with the legal suffix removed (Acme Cloud Services, Inc. → ACME CLOUD SERVICES). - Invoice number: INV- followed by the number without leading zeros. - Date: ISO 8601, and numeric dates on supplier documents are day-first. - Total: tax included, computed when a document only gives a subtotal and a VAT rate. - Currency: the ISO 4217 code, never the symbol. - Payment terms: whole days (Payable within two weeks → 14). - Category: one of five ledger labels, chosen by what was bought, not who sold it. Figure 2. A real held-out task. The vendor's name says "Travel", but the line item is docking stations, so the ledger category is hardware. The documents are generated, which is what makes the study clean: we control the ground truth exactly, so the ceiling is known and every miss is attributable. They vary layout, date formats, currency symbols, number padding, legal suffixes and whether the total is stated or has to be computed, and every document names the buyer (Contoso) as a distractor. ### A verifier that checks the record, not the prose Reward is computed field by field: each of the seven fields that exactly matches the house record earns 1/7. An episode counts as fully correct only if all seven match. There is no partial credit for a plausible-looking value — Office Supplies is not office_supplies. That strictness is the point: it is what a downstream accounting system would enforce. ### A learner that changes words, not weights The policy is the frozen model plus a playbook: a short list of lessons placed in its instructions. Each learning step runs this loop (Figure 1): - Sixteen training invoices, four attempts each. - The verifier scores every attempt. - Reflection: for each invoice where attempts scored differently — or all failed — the model compares the best attempt with the worst, reads the reference record the way one reads a grader’s feedback, and proposes general lessons. - A curator merges them into the playbook with explicit add, edit and delete operations, capped at 20 lessons so the prompt can’t grow without bound. - A gate runs the old and new playbooks on 16 fresh training invoices. The update is kept only if the score doesn’t drop. Figure 1. The learning loop. Only the playbook changes; the held-out invoices never enter it. If this resembles GRPO, that is intentional. Sampling a group of attempts per task and learning from how they differ is the same group-relative idea; here the “advantage” is written in language and applied to a playbook instead of a gradient. ### Controls that make the number mean something - Held-out split: 100 invoices scored before and after learning, never shown to the learner. - Control arm: the unchanged model evaluated twice, which measures run-to-run noise. - Oracle arm: the model given the written conventions and the full category mapping, which measures the ceiling. - Statistics: per-invoice paired differences, a bootstrap 95% confidence interval and a sign-flip permutation test. ## Experiments Modelgpt-5.4 via Azure OpenAI, default sampling Training96 invoices, one epoch: 6 steps of 16 invoices × 4 attempts Gate16 fresh training invoices per update Playbookat most 20 lessons; the reflector is the same gpt-5.4 Evaluation100 held-out invoices, one attempt each FrameworkLLM-RLE, our environment framework; every run is a configuration file plus a seed ### Control and ceiling Evaluated twice, the unchanged model scored 0.583 both times (95% interval of the difference −0.010 to +0.011). Anything larger than about a hundredth is signal. It already read amounts (99%) and payment terms (100%) almost perfectly, and failed exactly the convention fields: vendor 0%, category 19%, invoice number 21%. Not one of the 100 invoices was fully correct. Given the rules in its prompt, the same model scored 0.994, with 96% fully correct. So the gap is conventions, not reading ability, and the ceiling is high. ### Reflection v1: real gains, then a plateau The first learner made fast, real progress. Within two steps it learned the vendor convention (0% → 100%) and date and currency formats, and the gate rejected two updates that would have made things worse. Then training reward flattened at about 0.73. Invoice numbers and categories barely moved, and only 3 of 384 training attempts were fully correct (Figure 3, grey). Figure 3. Training reward per step. v1 plateaus with invoice number and category stuck; v2 solves both within two updates. ### The diagnosis: lessons that read well but can’t be followed We read the training attempts mid-run, before the held-out result existed. Two patterns stood out. Categories were semantically right but formatted wrong: Hardware, Office Supplies. Invoice numbers had their leading zeros stripped but no INV- prefix. The playbook explained why. The reflector had seen the reference records, yet its lessons were hedged: strip leading zeros “only when examples indicate they are non-semantic”; apply the INV- prefix “when examples or task patterns support that canonical format”. Our reflection prompt had asked for “general rules, not facts about this task”, and the reflector obliged with abstractions conditioned on examples. But the model being taught never sees examples or references — it sees one invoice and its instructions. The lessons sounded thorough and gave it nothing to act on. Figure 4. The same conventions, learned twice. Orange marks conditions the model cannot check; blue marks exact formats and allowed values. It echoes our earlier finding that a signal can look excellent by one measure and still mislead. Here the lessons read like expertise; only the verifier showed they weren’t working. ### Reflection v2: one change We changed only the reflection and curation instructions. They now state that the agent sees nothing but the task, forbid conditional hedges, and ask for conventions as direct rules with exact formats and example values. Facts about a single invoice remain excluded. Configuration, seed and held-out set stayed the same. The first update alone lifted gate reward from 0.607 to 0.929. By the third step, all 64 training attempts were fully correct, so there was nothing left to contrast, and the remaining steps made only small category refinements. ## Outcomes Figure 5. Held-out result. Reflection v2 matches the oracle that was given the rules. ArmReward before → afterChange in reward [95% CI]Fully correctTokens used Control (no learning)0.583 → 0.583+0.000 [−0.010, +0.011]0% → 0%43k Reflection v10.586 → 0.864+0.279 [+0.254, +0.306]0% → 26%483k Reflection v20.590 → 0.997+0.407 [+0.386, +0.429]0% → 98%451k Oracle (rules in prompt)— → 0.994—— → 96%46k Both learners’ improvements are significant (p = 0.0002, the smallest value 5,000 permutations can report). Reflection v2 improved all 100 held-out invoices and made none worse. Figure 6. Every convention field reaches 100%. The two fields the model could already read, total and terms, stay where they were. The per-field view shows where the gap lived and that closing it cost nothing elsewhere: the fields the model already handled — total amount and payment terms — stayed at 98–100%. On the invoice from Figure 2, the same model, given the same request, went from two correct fields to seven: Figure 7. The same invoice before and after learning. Note the category: the vendor's name no longer misleads it. What remains wrong. Two of the 100 invoices still miss one field, both the total. One is a $1.00 arithmetic slip. The other is a one-cent difference caused by rounding order: the learned rule rounds subtotal × (1 + rate), while the house rounds the tax line before adding it. That is a genuine convention the six steps did not surface. What the learned artifact is. The whole result of learning is a nine-lesson playbook in a JSON file, such as “Normalize invoice_number to INV- … Example: Invoice #0001234 → INV-1234“. It can be read, reviewed, diffed and reverted like any configuration change, and swapped back out instantly. It isn’t perfect: the category lesson lists labels that don’t exist in this ledger (shipping, utilities), a sign that the taxonomy was inferred rather than known. What it cost. The full v2 run — baseline evaluation, 384 training attempts, gate checks, reflection and final evaluation — used about 450k tokens, once. The recurring cost is the playbook itself: about 590 extra prompt tokens per request (156 → 746). Average latency barely moved (2.24 s → 2.30 s per request), within the run-to-run variation the control showed. ## Conclusion What this shows. For a gap made of conventions, an environment with an exact verifier was enough to close it from reward, with no weight updates, and the gain held on invoices the learner never saw — at the level of a model that was simply told the rules. Two further lessons matter as much as the headline: - The learner’s design mattered as much as the loop. The same environment, model and budget produced 26% or 98% depending on whether lessons were hedged or concrete. Lessons have to be written for what the policy can actually see. - The discipline did real work. The gate blocked two regressions, the control showed the noise was about a hundredth, and the oracle gave the result a meaning. Without the verifier, v1’s playbook would have looked like success. What it doesn’t show yet. - It is one seed on one model. - Reflection v2 was evaluated on the same 100 held-out invoices as v1. The change was designed from training attempts and committed before v1’s held-out result arrived, but a fresh held-out set still has to confirm it. - Reflection read the reference records, like a grader’s feedback. Learning from the reward score alone is the stricter test. - The environment is synthetic and its conventions are perfectly consistent; real documents are not. - One round of learner redesign was done by people, and that effort belongs in the cost. Where this goes next. This was the weight-free half of the question. The same environment and verifier now serve as the training signal for the weight-based half: GRPO on an open base model, scored by exactly the same checks, so the two approaches can be compared directly. Before that, we will replicate v2 across three seeds on a fresh held-out set, run reward-only reflection, repeat on a smaller model, and move from synthetic invoices to a real use case. If your team has a task governed by conventions like these, bring it to us. All notes --- ## You can’t trust a patch without a certificate — Perception URL: https://perception.club/notes/you-cannot-trust-a-patch-without-a-certificate Note · 2026-06-16 # You can’t trust a patch without a certificate We set out to answer a practical question: when you repair a trained model with a small patch — a LoRA adapter that fixes a failure mode, an edit that corrects a behaviour — how do you know the patch is safe to ship? Our answer is that the unit of trust is not the patch but the certificate: a per-dimension, pass/fail record of what the patch did and did not do, scored honestly, with a proof that the record survives being installed next to other patches. An earlier note reported the first half — that honest measurement matters, that a patch with a perfect training-loss curve can be quietly harmful, and that naively combining two good patches destroys both. This note reports the concluded study, now written up as a working paper. ## A certificate is only as good as its thresholds — so we calibrated them A certificate that fails every patch is as useless as one that passes every patch. Our first version set the locality threshold — how much a patch is allowed to disturb the capabilities it was not meant to touch — by fiat, at a round number. It rejected patches on what turned out to be measurement noise. So we replaced the guess with a rule, fixed before we collected any data: measure how much each probe moves under perturbations that carry no capability intent (random weight deltas, and re-runs of the same recipe at different seeds), and set the threshold just above that noise floor. Two things fell out of the calibration. First, the damage a bad merge does is structured, not a matter of size: a random weight change as large as a real merge is harmless, while the real merge at that same magnitude is catastrophic — so the harm lives in specific learned directions, not in the size of the change. Second, what had looked like a random “seed lottery” in a patch’s side effects was a small-sample illusion; measured at adequate resolution, the cost of SQL training to instruction-following is systematic, about −0.07, every time. That forced a design rule we now apply everywhere: a threshold of 0.02 is meaningless on a 50-item probe, where a single flip is already 0.02 — you need at least 200 items before the verdict is about the patch rather than the probe. With calibrated thresholds, the certificate became satisfiable for the first time. The remaining failures are real, not artefacts. ## Low-rank patches reach the edge of certifiability, but land on it With a calibrated, properly-resolved certificate, we hit a wall — and it was an informative one. Strong patches cleared the efficacy bar comfortably but kept failing locality on that same systematic instruction-following regression. The bottleneck was not the certificate; it was the patch mechanism. A low-rank update, at the scale we tested, is not a surgical edit — it reshapes the whole behaviour at once, so a real target gain comes bundled with collateral movement. But the certifiable corner exists. A deliberately gentle patch reached efficacy +0.20 with locality drift collapsed to exactly the threshold, and missed a clean pass on one remaining dimension by a floating-point epsilon — four flips out of two hundred. It is one recipe tweak away from a certified pass. We did not claim it: it was pre-registered as a final attempt, the boundary failure was caused by a real bug, and we fixed the bug forward only — the recorded near-miss keeps its recorded FAIL. No certified patch is claimed in this study. The honest statement is that certification is marginal, not impossible, with this class of patch. ## The conflict between two patches is readable from their weights When two patches destroy each other on merge, can you predict it in advance — cheaply, from the weights alone, without the expensive step of installing and re-testing? Largely, yes, and the mechanism is not the obvious one. We had predicted that patches writing to the same subspace would collide; the data reversed it. The governing variable is norm dominance: a merge is safe only when one patch is much larger than the other, so the big one swamps the small one harmlessly. When two patches are of comparable magnitude they collapse each other — even two patches for the same capability. “Same capability is safe” is false. We report this predictor as directionally strong but formally inconclusive by our own pre-registered bar — nine patch pairs is too few to fit the model cleanly — and we name the missing ingredient (an explicit norm-dominance term) for the follow-on. ## The result that settles installation: route, don’t merge If merging two patches is dangerous, what should you do instead? Keep them separate and serve each capability with its own patch — routing. Across every pair we tested, routing beat naive merging on 8 of 9, by a wide margin, and never lost. Naive merge is strictly dominated: there is no case where combining the weights wins. We had built a cleverer policy that decides per-pair whether to merge or route; it turned out to have no job to do, because routing is always at least as good. The optimal install policy collapses to one line — always route, never merge — unless a hard deployment constraint forbids keeping multiple patches around, in which case the predictor above earns its keep by refusing the catastrophic merges. Worst-case served-capability accuracy for each patch pair under naive merge (red) versus routing (green). Routing is at least as good on every pair and strictly better on eight of nine — naive merge is dominated. ## Does it generalise? A second scale, and a second domain Two natural objections to everything above are “that’s a quirk of one small model” and “that’s a quirk of SQL.” The concluded study answers both. First, the gap between honest measurement and the inflated kind is not a vibe — it is a near-deterministic line. The amount teacher-forced scoring overstates a model equals, almost exactly, how much room the model had left to fail (its headroom); the same line fits a model and one five times its size. So inflated scoring misleads precisely in the fallible regime where you would actually want to repair a model, and tells the truth only once the model has nothing left to get wrong — which is exactly when you would not bother patching. The teacher-forcing overstatement is a near-linear function of free-generation headroom (R² = 0.99), and the same line holds across a 5× range of model sizes. The two metrics agree only at task ceiling. Second, we took the entire machinery to an unrelated capability — generating tool and function calls — scored on an external, recognised benchmark rather than our own. The framework behaved identically. Two patch attempts were both refused: one made its own target worse, the other broke even, and the certificate additionally caught it dropping a capability it had nothing to do with — instruction-following — by 0.094. Tellingly, that same victim, instruction-following, was degraded by both the SQL training and the tool-call training: two unrelated repairs, the same collateral damage, exactly the kind of effect a single accuracy number hides and a per-dimension certificate surfaces. The framework is not SQL-shaped. The certificate in a second domain (tool-calling, scored on BFCL). Left: neither patch beats the base on its own target. Right: the break-even patch still leaks, dropping instruction-following past the locality threshold — a FAIL on two dimensions at once. ## What the study settled, and what it left open At every step, the certificate did its job: it refused a patch that every conventional signal endorsed, it caught the composition collapse, it was made trustworthy by calibration, and it held the line at the boundary rather than being quietly tuned to pass. The validated contribution is the instrument. What remains genuinely open — and is now a precise question rather than a vague hope — is whether any cheap patch mechanism can produce a fully certified repair. Low-rank adaptation reaches the corner and lands on its edge. A note on method, because it is part of the result. Every threshold and success criterion was written down before the run that tested it; we recorded a prediction the data reversed, and a near-miss that stayed a failure. A certification framework whose verdicts can be massaged at the boundary certifies nothing — so the discipline is not adjacent to the work, it is the work. The measuring-stick law spans two model scales (1.5B and 7B) and certification now covers two domains (text-to-SQL and tool-calling); the calibration and composition results remain at the 1.5B scale on SQL, and cross-domain composition is the named next step. This is a v1 working paper — findings are reported with their confidence intervals, refutations, and limitations intact. All notes --- ## Certified capability repair: what we’ve learned so far — Perception URL: https://perception.club/notes/certified-capability-repair-first-results Note · 2026-06-11 # Certified capability repair: what we’ve learned so far Our research question is whether a trained model can be fixed without retraining it. The first thing we learned is that the patch is not the product — the certificate is. A patch is a claim ("this edit improves capability X and touches nothing else"); a certificate is the measured evidence for that claim, dimension by dimension: did the target capability actually improve, did the declared neighborhood stay put, did general ability regress, and what did the patch cost. This note reports what happened when we built that machinery and pointed it at real patches. Setup, for the record: all experiments run on a pinned small open model (Qwen2.5-1.5B-Instruct, revision-locked), on the NL-to-SQL domain (Spider), scored by execution accuracy under greedy free generation, with bootstrap confidence intervals on everything. Verdicts are pass/fail per dimension, never averaged. ## Finding 1 — The standard way of measuring capability flatters the model Before trusting any repair measurement, we tested the measuring stick. Teacher-forced per-token agreement — the regime most training metrics live in — scored our base model at 0.948 on SQL generation. Free generation, where the model must produce the whole query unaided, scored 0.850 on the same probes. The gap is statistically significant, and the audit showed why: free generation fails by inventing tables, injecting spurious JOINs, and misusing aggregates — derailments that teacher forcing structurally cannot exhibit, because the gold token is fed back at every step. Any repair program that evaluates with teacher forcing is grading itself on a curve. ## Finding 2 — Training loss says nothing about whether a patch is safe We trained a LoRA patch to improve JOIN queries. By the usual signal it was a clean success: training loss fell smoothly from 0.55 to 0.17. By that signal alone, you would ship it. The certificate refused it on three dimensions: the target capability barely moved (+0.027, not significant), the neighboring capabilities collapsed (non-join SQL down 18 points), and general ability regressed. The audit pinned the mechanism: 72 of 73 newly-broken non-join queries failed because the patch had learned to inject JOINs everywhere. The patch made the model see joins in every question. Three iterations later, with mixed training data and more capacity, we reached a patch with a statistically significant target gain (+0.08). But the certificate still refused it, and the reason matters: the gain was not isolated. The whole SQL neighborhood moved together. Our conclusion from this arc is blunt — low-rank adaptation gives significant but non-isolated repair. It is a certifiable mini-fine-tune, not a precision edit. Genuinely surgical repair needs a different patch class, which is now an open line of work in the program. ## Finding 3 — Composing two good patches destroys both The enterprise version of this problem is never one patch; it is a registry of them. So we certified two independent patches — one for joins, one for aggregation (the aggregation patch was genuinely strong: +0.25, significant) — and installed both by the naive method, summing their weight deltas. The result was not degradation but destruction: both capabilities fell below the unpatched base model. The aggregation capability went from 0.85 patched-alone to 0.53 merged — worse than the 0.60 it started at. A certificate, we now know from measurement rather than argument, is not preserved under naive composition. ## Finding 4 — The obvious explanation for that destruction is wrong We expected the two patches to collide because they write to overlapping subspaces of the weights. Measured, the overlap is low — about 2.4× what random chance would give, with near-zero directional alignment — yet the merge was catastrophic anyway. The naive subspace-collision story is refuted. The live alternative is a magnitude effect: the merged perturbation is simply too large, pushing the model out of its working region regardless of direction. We have a cheap experiment queued to separate the two, and the answer reshapes how conflict between patches should be predicted. ## Where this leaves the program Zero patches have passed certification so far — and that is the system working, not failing. Every refused patch was refused for a measured, audited, mechanistic reason that a training-loss curve would have hidden. The near-term work follows directly from the findings: calibrating locality thresholds against placebo and seed-variant baselines (pre-registered before the data is collected), learning to predict patch conflict from weight-space features, and testing install policies that beat naive merging. The long-term shape is unchanged: input, failure examples; output, a certified patch — with proof it survives installation next to its neighbors. Results above are from a single 1.5B model on the SQL domain, with sample sizes of 40–150 per slice; composition findings are from one patch pair. We treat them as directional until replicated at 7B and on a second domain — both queued. All notes