ADR-0121: Semantic citation verification — does the cited article say what the rule claims?

Date: 2026-07-22 Status: Accepted Deciders: Adrian (Soft4U BV), Claude (Opus 4.8) Extends: ADR-0119 (structured citation binding)

Context

ADR-0119 made citations referentially sound: a rule can no longer name an instrument nobody mapped, an article that does not exist, or render as grounded when it is not. A contract test enforces this on every PR.

It does not make them correct. The proof is direct — every one of the 15 mis-citations corrected on 2026-07-21 passes the ADR-0119 gate cleanly today:

Wrong citation	Resolves?	Why it was wrong
AMLR Art. 28 (for CDD)	✅	Art. 28 is the RTS-delegation article, not the CDD duty (Art. 22/34)
AMLR Art. 35-37 (for PEP)	✅	Those are third-country countermeasures; PEP is Art. 42-44
Czech AML Act s. 10 (for PEP)	✅	s. 10 is "Identification by an Intermediary"; PEP is s. 9a
Czech AML Act s. 29 (for BO)	✅	s. 29 is "Financial Postal Services"; BO is s. 9(2)(b)
AMLR Art. 62-63 (for BO discrepancy)	✅	Art. 24 is the discrepancy duty

All name real articles. The gate asks "does this exist?" and never "does it say this?". Those are different properties, and the second is the one an officer, an auditor and a regulator actually rely on.

That audit was an LLM reading each article's text against each rule's purpose. It was manual, one-off, and nothing automated replaced it — so the class it found can silently return the moment a rule is added or edited.

Why this matters beyond tidiness. Under EU AI Act Art. 12 and AMLR, an AI decision must be traceable to its stated basis. A legal basis that names the wrong provision is worse than a missing one: it is plausible, it survives review, and it appears in the decision memorandum and the regulator case-pack. The project's cardinal doctrine — add scrutiny, never suppress a signal — is violated quietly when the scrutiny is attached to the wrong law.

Decision

Add a semantic citation verifier: for each rule, retrieve the cited article's actual text and judge whether it supports the rule's stated purpose. Reuse the ADR-0111 finding-correctness pattern rather than inventing a second shape for the same idea.

1. Four verdicts, not a boolean

SUPPORTS · CONTRADICTS · UNSUPPORTED · INDETERMINATE

matching CorrectnessVerdict in finding_correctness_verifier. The distinction that does the work is UNSUPPORTED vs CONTRADICTS: "this article does not mention PEPs at all" is a different defect from "this article says the opposite", and they have different fixes. INDETERMINATE is reserved for we could not judge — never used to mean probably fine.

2. Judge the LEGAL BASIS, not verbatim overlap

Amended 2026-07-22 after measuring the first run. The prompt originally used pure adversarial framing — "refute the citation; default to UNSUPPORTED when uncertain". Run against all 92 real citations it returned 7 supports and 62 unsupported, rejecting citations such as AMLR Art. 34 for an enhanced-due-diligence rule because Art. 34 does not enumerate that specific risk factor — when Art. 34 is the EDD duty.

That is this ADR's own predicted failure, arriving immediately: a false-rejection rate that would get the control switched off. The error was conflating two questions — "is this the right legal basis for this obligation?" (what we want) with "does this article state this fact verbatim?" (far too strict).

v2 asks the first. It accepts an article that creates, defines or governs the duty the rule invokes, even when the rule's trigger is not enumerated there — a rule supplies the risk assessment, the article supplies the duty. It rejects an article that governs a different subject, which is the real defect: Art. 28 merely instructs AMLA to draft standards and imposes no duty on an obliged entity. The worked example and its reason are both in the prompt, because an example without its reason teaches nothing transferable.

Re-run: 41 supports, 29 unsupported, 22 indeterminate. A targeted check on six known cases — the three historical mis-citations plus the three v1 falsely rejected — was 6/6 correct in both directions.

The prompt id moves v1 → v2 with the change, so a stored verdict can never misattribute which prompt produced it.

3. Fail-closed, and scoped to what fails

An INDETERMINATE or unavailable judge blocks the citation, never passes it. But the blocking is per-citation: one unjudgeable rule does not fail the whole suite, because a check that takes down CI on an API hiccup gets disabled.

4. Two tiers, because cost and coverage pull apart

• Diff-aware (CI, every PR): judge only citations whose regulatory_basis is new or changed in the diff. This is where mistakes are actually introduced, and it is a handful of LLM calls. Live — the free baseline gate runs in the normal test suite, so a new or edited citation fails until someone runs the audit script and commits the verdicts.
• Full sweep (scheduled): catches drift the diff cannot see — the corpus is re-ingested, an article is renumbered, a law is amended, the judge's prompt is revised. This is what turns the 2026-07-21 audit into a standing control.

Amended 2026-07-22 — the scheduled tier as built. A system citation-drift-sweep Temporal Schedule (the ADR-0096 pattern, dark-launched citation_drift_sweep_enabled=False) fires CitationDriftSweepWorkflow every citation_drift_sweep_interval_hours. Two properties are load-bearing and neither was obvious when this ADR was first written:

1. The sweep never writes the baseline. A scheduled job that refreshed its own reviewed record would auto-bless the exact drift it exists to catch — §7's rule, applied to the control itself. It reports; a human runs scripts/audit_citations.py --all and commits.
2. Staleness is decided deterministically, before any LLM runs. A recorded verdict is stale iff the article's content_hash moved or the recorded prompt_id differs from the current one. Judging is temperature=0, so re-judging an unchanged article under an unchanged prompt re-asks a question whose answer is already stored. Only the genuinely stale subset reaches the judge. Measured against the committed baseline on 2026-07-22: 42 accepted + 50 declared = 92, every row carrying a hash and the current prompt id — so a sweep today costs zero LLM calls and still reports honestly. That is what makes a nightly control affordable rather than theoretical.

A re-judged citation that still returns SUPPORTS is still reported: the stored verdict refers to text that no longer exists, so the record is stale even though the citation is right. Reporting only regressions would let the register drift out of correspondence with the corpus while looking clean.

Everything that is not "verified fresh" is drift — an unreviewed citation, a vanished article, an unreadable baseline, a harvest that yields zero. The one deliberate exception is the declared exceptions block: it is already a known, ratcheted, visible backlog, and re-alarming it nightly would bury real drift under a permanent alarm nobody reads. It is still counted in every report.

Live state: GET /monitoring/citation-drift (super_admin) reports the flag, the schedule's actual presence, the baseline's real contents and how many verdicts sit under an older prompt. A disabled flag, a missing schedule or an unreadable baseline is healthy=false with a status_note — never silent-green.

5. A baseline register, ratcheted

The baseline works like a lockfile, and the split is what makes the control survivable: a free, deterministic test asserts every registry citation appears in the baseline with the same text, and the LLM runs only when a human is deciding something. A control requiring an API key and real spend on every commit is one that gets disabled.

Rejections are recorded in a separate exceptions block with the judge's reasoning, and a ratchet test forbids that block from growing. Blocking all work until every flagged citation is re-authored would get the control switched off; hiding the rejections would make the audit a lie. A declared backlog that cannot grow is the honest middle — the same shape as KNOWN_CORPUS_GAPS and the duplicate-article ratchet.

First run (2026-07-22): 42 accepted, 50 declared — 22 that do not resolve at all (already known from ADR-0119) and 28 genuine findings, including a case no referential check could ever catch: nl_psp_pep_match cites Wwft Art. 8(5)(a)-(c) for PEP obligations, and that paragraph now governs crypto-asset services — the law was amended and renumbered under a citation that still resolves.

Existing citations that a judge accepts are recorded with the verdict, the article's content_hash, the model and the prompt id. A citation whose article text changes is re-judged automatically — that is the drift detector, and it is why the hash is part of the record rather than decoration.

Rejections are declared in a register with the judge's reasoning, exactly as KNOWN_UNGRAMMATICAL / KNOWN_CORPUS_GAPS are, and checked in both directions: an entry that starts passing must be removed.

6. Determinism and traceability

temperature=0; the prompt comes from the registry (ADR-0026) so it is versioned; the model tier is recorded (ADR-0029). Every verdict carries (rule_id, citation, article_content_hash, model, prompt_id, verdict, reasoning). That tuple is what makes the control auditable under Art. 12 rather than an opaque assertion that something was checked.

7. It never edits a citation

The verifier reports; a human fixes. An automated corrector would be a system quietly rewriting its own legal basis — the least auditable thing this codebase could contain.

Consequences

Positive

• The defect class that produced 15 mis-citations becomes a standing control rather than a memory of one afternoon.
• Corpus drift is detected: re-ingestion that changes an article's text re-opens every citation resting on it.
• The verdict record is exactly the traceability Art. 12 asks for.
• Diff-awareness puts the feedback at authoring time, when the fix is cheap.

Negative

• A judge can be wrong in both directions. A false CONTRADICTS blocks a correct citation and, if that happens often, the control gets switched off — the failure mode this project has already reasoned about for gates. Mitigated by the register (a human overrides with a recorded reason) and by adversarial-but-conservative prompting, not eliminated.
• LLM cost and latency; hence two tiers rather than one.
• The register is hand-maintained. Mitigated by bidirectional checking and by hash-triggered re-judging, so it cannot silently ossify.
• Judging a citation against an article whose corpus text is wrong yields a confident wrong verdict. ADR-0118's gate is the upstream guard; this ADR assumes the corpus contains the law it claims to.

Neutral

• No rule, severity, score or escalator changes. This governs the legal basis label and its correctness only.

Alternatives Considered

Alternative 1: Deterministic keyword matching (no LLM)

Check that the article text contains terms from the rule's category ("politically exposed" for a PEP rule). Rejected: it is exactly the "free-text keyword guessing" that classify_sanctions_record documents as forbidden, because it misclassifies both ways — Art. 28 mentions customer due diligence while being the delegation article, and a correct article may use different vocabulary entirely. It would produce confident nonsense.

Alternative 2: Verify at runtime, per finding

Judge the citation when a finding is produced. Rejected: it puts an LLM call on the investigation path for a property that is a static fact about the rule registry, not about the case. The same citation would be re-judged thousands of times, and a slow judge would degrade case processing.

Alternative 3: Full sweep only, no diff tier

Simpler. Rejected: the feedback would arrive days after the rule was written, when the author has moved on — the same reason the ADR-0119 contract test runs per-PR rather than monthly.

Alternative 4: Do nothing; rely on periodic manual audit

This is the status quo that produced the 15. Rejected explicitly.

Decision context

• Latency: zero on the request path (Alternative 2 rejected for this reason). CI adds a handful of calls on changed citations only; the full sweep is scheduled.
• Dependency surface: none new — the existing PydanticAI agent stack and prompt registry.
• Debuggability: every verdict names the rule, the citation, the article hash and the judge's reasoning, so a disputed result is inspectable rather than an opaque "failed".
• Reversibility: hours — a config flag, dark-launched, as ADR-0111 was.
• Blast radius: additive. No existing code path changes behaviour; the verifier only reports.
• Alternative considered: deterministic keyword matching (Alternative 1) — rejected because it is the keyword-guessing anti-pattern the sanctions classifier already forbids for exactly this reason.

References

• ADR-0119 (structured citation binding) — the referential half
• ADR-0111 (finding correctness re-verifier) — the pattern reused
• ADR-0026 (prompt registry), ADR-0029 (model tiers), ADR-0067 (fail-closed)
• docs/reviews/2026-07-21-citation-accuracy-audit-calibration-review.md — the manual audit this makes standing