Three times out of three, it refused a free change

Finding

The control arm refused a change that was always free, three times out of three. The change equalised a noise value across the two themes. That value is a judgement about how noise reads on a dark page, and nothing fails if the two match. But it sits in a build checklist as a bare imperative, between two items that really are load-bearing. Every run read it as a measurement.

So an under-annotated specification does not only let bad changes through. It makes a careful reader refuse changes that cost nothing — confidently, and for reasons that sound right. That is a different failure from the one I went looking for. Nothing about it looks like a mistake, which is what makes it hard to find.

The notation earned nothing. That was the question I actually asked — whether the medieval form for carrying provenance does work a plain sentence does not. As ordinary prose, the facts took the score from five of nine to eight of nine. As sigla and a witness legend they added one further point, from one run. That point dissolves on inspection.

Three replicates an arm. The second result rests on a single run, on the one item whose rubric is loose. Read the first finding as a failure mode worth looking for, the second as an absence of evidence. At this size, that is not much of an absence.

Why I looked at this

Not a survey of quaint old formats. Chapbook is a small CSS system I maintain, and its README calls a chapbook a distribution technology rather than a look — standard formats, shared stock, no design per title. That is not a claim about taste — it is a claim about economics, and it generalises.

Printing was cheap and design was expensive, so design became shared stock. Generation is now free and deciding is expensive. So every old form that existed to spare someone a decision is live again. That is a search procedure rather than a list.

You are looking for a form whose constraint matches a constraint you have now — usually because the thing that was scarce then is scarce again, for a different reason. Three worked examples, none of them tested.

Locke’s commonplace index of 1685 files a heading under its first letter plus the first vowel after it. Fussy, until you see the constraint. A bound book is append-only and cannot be reordered, so the index has to absorb all the structure. Agent memory files are append-only for the same reason.

Ogilby’s strip maps of 1675 chart one route as a ribbon and show only what you pass. Useless for anything else — far better than a real map for that journey. Context packs have the shape, and everyone ships the territory when the task wants the route.

The Nautical Almanac’s human computers of 1767 were unreliable and paid by the piece. So every value was computed twice, by two people who did not know each other, and a third — the comparer — reconciled them. Multi-agent verification, arrived at empirically under cost pressure.

The form I went after is the apparatus: the machinery around a text that carries its provenance. A lemma, a siglum, the variant readings, compressed into notation at the foot of a scholarly edition.

The claim it was meant to test is about now. Documents hold human text, model text, quoted text and unverified claims, all rendered identically. Provenance is attached at the document level. That is the one level where it is useless. The question is never whether a page was AI-assisted. It is whether this sentence has a source.

The question

Does marking a specification with provenance help an agent read it, or does the notation for carrying it do work of its own — and if the notation does none, what is left worth building?

What I expected

That agents would improve load-bearing values into breakage, and that an apparatus would stop them.

What I tried

Marked the same specification up three ways, and scored the arms against answers written first.

Three arms rather than two, because two would have confounded the information with the notation. Arm A was the specification as it stood, five hundred and nineteen lines. Arm C added the provenance facts as ordinary sentences. Arm B carried the same facts as a witness legend and an apparatus band. A to C isolates the information; C to B isolates the notation, which was my actual claim.

Assembled the arms by inserting text at three fixed positions in a byte-identical base, so position and surrounding prose are controlled. Sonnet, three replicates an arm.

The task was a pull request with three proposed changes, each answered BLOCK or NEGOTIABLE. Replace :where() with :is() in the reset. Equalise --grain across the two themes. Round --faint from #666 to #767676. One of the three was meant to be permitted, so an arm that refuses everything cannot score well. The answers were written before any run. Every run was checked afterwards for a single tool call, which proves none of them reached the real stylesheet.

What happened

Five things, and the first is the one I did not predict.

The control arm blocked the harmless change every time. Zero of three. --grain is 0.028 in light and 0.02 in dark. The only thing the specification says about the pair is checklist item eight: Lower --grain in dark. Items seven and nine on either side of it are bolded, and both have checks behind them in the build. One of the three runs refused on reasoning I could not fault — not a misreading. I am going from memory of reading it. results.tsv records verdicts and nothing the runs said, so that judgement is mine and the record does not hold it.

The break I predicted happened once. One control run called the :is() swap a real but debatable tradeoff, not a proven break, and would have merged it. :is() takes the specificity of its most specific argument. So the reset scores (0,1,1), outranks the rhythm rule at (0,1,0), and every gap in the column collapses to zero. In every engine, with no console warning.

The information gained three points. Arm A scored five of nine, arm C eight of nine. Item one went from two of three to three of three, item two from zero of three to three of three. Item three moved the other way — three of three down to two — which the headline number absorbs rather than shows.

The notation gained one point, from one run, and it dissolves. Arm B scored nine of nine. The one arm-C miss answered NEGOTIABLE on item three while reasoning merge only with that check done, which is BLOCK’s substance under a looser label. Score it generously and C ties B. Formal sigla contribute exactly zero.

Scored on mechanism as well as verdict, the control arm drops to three of nine. results.tsv carries an item1_mechanism column the scoring ignores. Arm A’s two correct blocks are both recorded vague, and its third is wrong. Arms B and C are exact in all six runs. Demanding the reason as well as the verdict, the arms read three, eight and nine of nine. The gap widens from three points to five. That is an observation, not a result. The rubric was written before the runs and this re-scoring was not.

The round that measured nothing

The first round was null, and I built the null myself. Nine runs, zero of eighty-one token decisions changed, no variance anywhere. The task was a palette audit reporting two contrast failures, both false positives. Every arm correctly kept all nine tokens.

The extract I handed them stated the --rule exemption and every measured ratio in plain prose. There was no headroom. If your document already answers the question out loud, your eval measures reading comprehension rather than format. Round two moved to the full specification, with the facts in their natural sections and a task whose answer the document did not state.

Where it broke

The question I actually asked is answered by one run, on the softest item in the set. Three replicates an arm, all of it Sonnet, so only large effects were detectable. The A-to-C gap is large but not consistent per item — item three moves backwards. The whole C-to-B result is a single run on item three, whose rubric is loose enough that the task prompt carries a written caveat about it.

The arms were not length-matched, and note 003 ran a fourth arm for exactly this reason. Round two’s apparatus insert is a hundred and seventy words against the prose insert’s two hundred and four. Seventeen per cent fewer words for the same facts, and one point higher. Round one reverses the direction — a hundred and fifty-two against a hundred and eighteen. So length cannot explain arm B’s point. The honest reading is compression rather than comprehension. Same facts, fewer words, no worse result. That is a smaller claim than the one under test.

The verification re-run’s contamination check is not in the record. The round-two task prompt states that every run was verified at one tool call. verification-results.tsv carries no such column and no note. It also pointed the agent at a real file in a real repository, where wandering is easier than with a standalone arm file. The headline result and the verification are not held to the same standard.

The apparatus runs quoted the notation back, and I cannot interpret that. Round one’s cited_notation column records all three doing it, and no prose run doing it. But the decisions were identical. Per dated M witnesses in the reasoning is equally consistent with useful salience and with a model echoing available vocabulary, and nothing in either round separates the two.

The specification, the arms, the rubric and the checklist line that fooled the control arm are all mine. The failure I am reporting is one I wrote, scored against answers I wrote. That is what makes it a real case rather than a toy. It is also what makes it one case.

Current view

Provisional, and close to the reverse of what I set out to build.

I came in expecting to ship a notation — a witness legend, sigla, three CSS classes to render them. What shipped was sentences — no legend, no sigla, no band. The provenance went into Chapbook 1.2.2 as ordinary prose, and re-running the task against the committed file at addc105 scored eight of nine. The harmless-change item went zero of three to three of three. Its one miss was item three again — the same soft item.

The classes .lem, .sig and .app were written and rendered against real content in both themes, then deliberately left out. The system’s rule is that a component earns its place after two builds, and this had one.

What survives is narrower than an apparatus. If you’re writing a specification an agent will read, it is one line long.

Mark which lines are measurements and which are preferences. The cost of leaving that out is not only the bad change that gets through. It is the free change that gets stopped — by a reader doing its job carefully, with reasoning I could not fault.

Two rounds produced no evidence for formal sigla over plain sentences. At three runs an arm that is a weak absence, and compression is the only part of arm B’s result I would still defend.

What I’d try next

The cheapest one first, because it settles a caveat rather than opening a question.

Re-run round two with the inserts padded to the same length. The apparatus insert is thirty-four words short of the prose one. Nothing else changes, and the compression reading either survives or doesn’t. Then record tool_uses per run in the tables instead of asserting the check in prose. The verification re-run needs it most, and it is a column.

The direction I now think is stronger was never tested. A model has nowhere to put uncertainty except hedging prose — which degrades the sentence, is unquantified, and cannot tell I measured a range from I am guessing. That wants a different shape: generate first, then score in a second pass whether the uncertainty landed where it belonged.

Note 001 had a missing control, 002 a missing comparison and 003 a missing holdout — and 003 called the pattern stable enough to be the finding. This one has three arms, answers written before the runs, a contamination check on the headline result and a re-run against the shipped file. The pattern broke. What is missing now is not a control but the other half of the question. Both rounds tested reading, and nothing tested whether an apparatus helps a model write.