ADR-0167: Process rules get detectors — and where a rule cannot be enforced, the repo says so

Status: Accepted Date: 2026-08-01 Issues: #965, #966, #967, #968, #970 (methodology audit epic s4u-methodology#23)

Context

A methodology audit measured this repository's own process controls against merge history rather than against their documentation. Four of the most-cited controls turned out to be enforced by nothing:

Rule	Where it is stated	Measured state
Calibration Review Checklist (#965)	CLAUDE.md:201 "MUST pass … all 8 checks"; checklist line 12 "Reviewers MUST block on an unticked box"	0/40 recent merges retained the template section; 15/40 had no calibration section at all, several compliance-facing; every freehand rewrite sampled had dropped CR4
Tests must pass before PR merge (#967)	CLAUDE.md:34; methodology "ENFORCED … blocks the merge"	24 of the last 25 merges did not observe it: 21 merged before the gate finished, 3 merged after it was already red (#885 by 72 min, #880 by ~22 h). PR #891 merged with the ADR-0121 citation gate red
ADR register integrity (#966)	Three docs instruct running scripts/check-adr-register.sh; two published pages assert it runs "in CI" and is "Blocking (fail-closed)"	The file did not exist in this repo. Every documented invocation could only exit 127; one plan template wrapped it in 2>/dev/null ||, making "missing" and "failing" the same soft advisory
ADR gate registered (#970)	appendix-g-adr.md prescribes check-adr.sh as a Stop hook	Registered in no settings file. Not blocking, and not warning either — a file with no caller. The register had drifted 14 entries in CLAUDE.md, 13 in docs/adr/README.md

This is the project's own named worst defect class — claim vs check: an artifact reporting a control's SHAPE while nothing observes its STATE. It had landed on the controls that guard everything else, including the checklist that ADR after ADR cites as the reason a compliance-facing change is safe.

Decision

Build a detector per rule, as a discovery mechanism for the next instance rather than a patch for the ones found. Four things follow from that:

1. Applicability is decided by measurable state, never by the author. The Calibration Review job derives applicability from the DIFF (scripts/check_calibration_review.py, compliance-facing path tokens drawn from CLAUDE.md's own enumeration). Each of the eight checks carries a stable slug CR1…CR8 so a freehand rewrite still matches and a dropped check — the measured failure mode — is detectable. Each must be dispositioned: [x], or N/A — <reason ≥ 15 chars>. Slugs appearing only in an HTML comment or a fenced block do not count.

2. A tick is a self-assertion, and the repo says so out loud. The one exception is a claim measurable state can refute: CR5 ("adversarial fixtures") ticked while the diff touches no test file fails. Beyond that, no PR-body detector can prove a check is TRUE. docs/calibration-review-checklist.md now carries a table naming exactly what is mechanically enforced and what is not, rather than a heuristic that would read as evidence while being none.

3. Where prevention is unavailable, observe. At authoring time no mechanism could require a check here (classic protection 404, both ruleset endpoints []) — that is the state under which all 24 baselined violations accrued. It changed while this ADR's PR was open: ruleset master required checks was armed 2026-08-01 21:36:44+02:00, one minute before the enforcement-state job's first run, which caught it and went red. Backend Tests, Frontend Tests, Lint and Calibration Review are now genuinely required. Observation remains, for a reason recorded in enforcement-state.json: repository admins hold an always bypass and the owner is the only human actor, so a red gate can still be merged past — silently, unless something observes it. scripts/check_merge_gate_observance.py therefore measures each merge's timestamp against the required jobs' own completed_at/conclusion — the state at the moment of merge, which the retrospective green tick in gh pr checks does not report. Historical debt is DECLARED in docs/process-gates/merge-gate-baseline.json, not deleted; the baseline is a ratchet (an entry may only describe a PR merged at or before its as_of), so a new violation cannot be silenced by widening it.

4. The enforcement claim itself gets a detector. docs/process-gates/enforcement-state.json is the single authority for the sentence "what actually blocks a merge here", and the enforcement-state job compares it to the live API on every run. A doc that says "ENFORCED" is now contradicted by a checked-in, machine-verified file; and if protection is ever armed, the job fails until the declaration is updated — the change cannot land silently in either direction.

Supporting changes: scripts/check-adr-register.sh vendored from the kit with a provenance header; scripts/check_adr_register_integrity.py wrapping it and additionally checking the CLAUDE.md register the kit script never looks at; both registers backfilled (13 + 14 rows added, none deleted); check-adr.sh registered as a third Stop hook; scripts/check_hook_registration.py asserting every gate script has a caller or a declared reason not to; and the pre-push hook rewritten so its description matches its behaviour, with a # GATE: manifest the same detector verifies.

Consequences

Positive. The four rules are observable for the first time. The register detector found and fixed 27 real drift rows across two registers. Registering check-adr.sh restores the ADR-presence question at session end. The pre-push hook no longer reports a missing ruff binary as "lint errors found" (#964's shim defect, encountered while making the hook honest).

Negative, and load-bearing. Of the five jobs added here, exactly one (Calibration Review) is a required status check; the other four are advisory and must not be described otherwise. Even a required check is bypassable by a repository admin. Anyone citing any of this in a compliance or conformity artifact must read enforcement-state.json first, which is machine-verified against the live API on every run. The merge-gate baseline ships with 24 declared violations; that number is the repo's actual debt, and it should shrink.

A defect this change caused, and caught. Arming the ruleset with the context Calibration Review while the job reported as Calibration Review (compliance-facing diffs) would have blocked every PR permanently on a check that could never report — the "a gate that cannot run is indistinguishable from a gate that passed" failure, pointing the other way. The job was renamed to match the context exactly, and compare_enforcement now fails when a required context is produced by no job in .github/workflows/. The name is now a contract, and it is checked.

Also negative. The calibration detector's coverage is a path allow-list, so a genuinely compliance-facing change on a path nobody anticipated is not caught. Widening it is a data edit; narrowing it silently shrinks the gate, which is why the exemption list is three entries long and no longer.

Alternatives considered

Enable branch protection. Measured unavailable at authoring time (both endpoints empty) and recorded as the exit_condition in enforcement-state.json rather than left implicit — then done, by the repo owner, while this PR was open. The detector's value is that the transition was observed within a minute rather than assumed in either direction; the exit_condition now describes disarming instead.

Require prose next to each ticked check. Rejected. It would prove prose was typed, not that a review happened — precisely the shape-not-state failure this ADR exists to end. Shipping it would have made the control look stronger while measuring nothing new.

Delete the drifted register rows to make the script green. Rejected outright. The register is the only human-readable index of 159 decisions the compliance story cites; "passing" by deleting entries is the defect, not the fix.

Uncomment pytest in the pre-push hook (#968's first option). Rejected: a 20-minute push gate is a gate that gets disabled. The hook now runs what it can run fast and states plainly what it does not run and where that runs instead — the option the issue itself offers as the alternative to "the template's honest '(Optional)' and the canon's normative 'must' cannot both stand".

Fail the merge-observance job on the historical debt. Rejected: it would be red for the next 25 merges and then ignored — this repo's own scar (a gate that cannot run is indistinguishable from a gate that passed). The ratcheted baseline follows the ADR-0121 citation-lockfile pattern instead.