You can’t trust a patch without a certificate
We set out to answer a practical question: when you repair a trained model with a small patch — a LoRA adapter that fixes a failure mode, an edit that corrects a behaviour — how do you know the patch is safe to ship? Our answer is that the unit of trust is not the patch but the certificate: a per-dimension, pass/fail record of what the patch did and did not do, scored honestly, with a proof that the record survives being installed next to other patches. An earlier note reported the first half — that honest measurement matters, that a patch with a perfect training-loss curve can be quietly harmful, and that naively combining two good patches destroys both. This note reports the concluded study, now written up as a working paper.
A certificate is only as good as its thresholds — so we calibrated them
A certificate that fails every patch is as useless as one that passes every patch. Our first version set the locality threshold — how much a patch is allowed to disturb the capabilities it was not meant to touch — by fiat, at a round number. It rejected patches on what turned out to be measurement noise. So we replaced the guess with a rule, fixed before we collected any data: measure how much each probe moves under perturbations that carry no capability intent (random weight deltas, and re-runs of the same recipe at different seeds), and set the threshold just above that noise floor.
Two things fell out of the calibration. First, the damage a bad merge does is structured, not a matter of size: a random weight change as large as a real merge is harmless, while the real merge at that same magnitude is catastrophic — so the harm lives in specific learned directions, not in the size of the change. Second, what had looked like a random “seed lottery” in a patch’s side effects was a small-sample illusion; measured at adequate resolution, the cost of SQL training to instruction-following is systematic, about −0.07, every time. That forced a design rule we now apply everywhere: a threshold of 0.02 is meaningless on a 50-item probe, where a single flip is already 0.02 — you need at least 200 items before the verdict is about the patch rather than the probe. With calibrated thresholds, the certificate became satisfiable for the first time. The remaining failures are real, not artefacts.
Low-rank patches reach the edge of certifiability, but land on it
With a calibrated, properly-resolved certificate, we hit a wall — and it was an informative one. Strong patches cleared the efficacy bar comfortably but kept failing locality on that same systematic instruction-following regression. The bottleneck was not the certificate; it was the patch mechanism. A low-rank update, at the scale we tested, is not a surgical edit — it reshapes the whole behaviour at once, so a real target gain comes bundled with collateral movement.
But the certifiable corner exists. A deliberately gentle patch reached efficacy +0.20 with locality drift collapsed to exactly the threshold, and missed a clean pass on one remaining dimension by a floating-point epsilon — four flips out of two hundred. It is one recipe tweak away from a certified pass. We did not claim it: it was pre-registered as a final attempt, the boundary failure was caused by a real bug, and we fixed the bug forward only — the recorded near-miss keeps its recorded FAIL. No certified patch is claimed in this study. The honest statement is that certification is marginal, not impossible, with this class of patch.
The conflict between two patches is readable from their weights
When two patches destroy each other on merge, can you predict it in advance — cheaply, from the weights alone, without the expensive step of installing and re-testing? Largely, yes, and the mechanism is not the obvious one. We had predicted that patches writing to the same subspace would collide; the data reversed it. The governing variable is norm dominance: a merge is safe only when one patch is much larger than the other, so the big one swamps the small one harmlessly. When two patches are of comparable magnitude they collapse each other — even two patches for the same capability. “Same capability is safe” is false. We report this predictor as directionally strong but formally inconclusive by our own pre-registered bar — nine patch pairs is too few to fit the model cleanly — and we name the missing ingredient (an explicit norm-dominance term) for the follow-on.
The result that settles installation: route, don’t merge
If merging two patches is dangerous, what should you do instead? Keep them separate and serve each capability with its own patch — routing. Across every pair we tested, routing beat naive merging on 8 of 9, by a wide margin, and never lost. Naive merge is strictly dominated: there is no case where combining the weights wins. We had built a cleverer policy that decides per-pair whether to merge or route; it turned out to have no job to do, because routing is always at least as good. The optimal install policy collapses to one line — always route, never merge — unless a hard deployment constraint forbids keeping multiple patches around, in which case the predictor above earns its keep by refusing the catastrophic merges.
Does it generalise? A second scale, and a second domain
Two natural objections to everything above are “that’s a quirk of one small model” and “that’s a quirk of SQL.” The concluded study answers both. First, the gap between honest measurement and the inflated kind is not a vibe — it is a near-deterministic line. The amount teacher-forced scoring overstates a model equals, almost exactly, how much room the model had left to fail (its headroom); the same line fits a model and one five times its size. So inflated scoring misleads precisely in the fallible regime where you would actually want to repair a model, and tells the truth only once the model has nothing left to get wrong — which is exactly when you would not bother patching.
Second, we took the entire machinery to an unrelated capability — generating tool and function calls — scored on an external, recognised benchmark rather than our own. The framework behaved identically. Two patch attempts were both refused: one made its own target worse, the other broke even, and the certificate additionally caught it dropping a capability it had nothing to do with — instruction-following — by 0.094. Tellingly, that same victim, instruction-following, was degraded by both the SQL training and the tool-call training: two unrelated repairs, the same collateral damage, exactly the kind of effect a single accuracy number hides and a per-dimension certificate surfaces. The framework is not SQL-shaped.
What the study settled, and what it left open
At every step, the certificate did its job: it refused a patch that every conventional signal endorsed, it caught the composition collapse, it was made trustworthy by calibration, and it held the line at the boundary rather than being quietly tuned to pass. The validated contribution is the instrument. What remains genuinely open — and is now a precise question rather than a vague hope — is whether any cheap patch mechanism can produce a fully certified repair. Low-rank adaptation reaches the corner and lands on its edge.
A note on method, because it is part of the result. Every threshold and success criterion was written down before the run that tested it; we recorded a prediction the data reversed, and a near-miss that stayed a failure. A certification framework whose verdicts can be massaged at the boundary certifies nothing — so the discipline is not adjacent to the work, it is the work.
The measuring-stick law spans two model scales (1.5B and 7B) and certification now covers two domains (text-to-SQL and tool-calling); the calibration and composition results remain at the 1.5B scale on SQL, and cross-domain composition is the named next step. This is a v1 working paper — findings are reported with their confidence intervals, refutations, and limitations intact.