Skip to main content

ADR-0119: Structured citation binding between rules and the Lex corpus

Date: 2026-07-22 Status: Accepted Deciders: Adrian (Soft4U BV), Claude (Opus 4.8)

Context​

The regulatory knowledge layer has two halves that meet in the middle.

The corpus half is well-formed. corpus_config.py declares each instrument; ingest.py runs fetch → parse → chunk → embed → index into lex_regulations (short_name, jurisdiction, celex/eli, source_url, content_hash) → lex_articles (article_number, article_title, full_text, hierarchy_path) → lex_chunks. Identifiers, provenance, versioning and hashes are all present, and ADR-0118's Wave-0 integrity gate now asserts the stored corpus contains law.

The rules half is string-valued. reasoning_template_registry.py carries rules whose regulatory_basis is free text:

"EU-AMLR Art. 10 §4(b) — risk factors including nature of business relationship"
"GwG s. 43 (Meldepflicht — SAR obligation to FIU)"
"Czech AML Act s. 9a"

When a rule fires, that string is copied onto the finding and travels into the case record, the decision memorandum PDF, the regulator case-pack export (ADR-0069) and the officer UI.

Nothing joins the halves. There is no foreign key from a rule or a finding to lex_articles.id. The linkage is re-derived at read time by three independent parsers of the same string:

ConsumerResolution mechanism
RegPill.tsxregex families (REG_REF_REGEX, FULL_CITATION, splitCitationGloss)
LexQueryService.get_article()WHERE short_name=:reg AND article_number=:num LIMIT 1
citation_verifier.pyits own path

The citation is a convention, not a constraint. Rules say GwG s. 43; the corpus stores short_name='DE-GwG', article_number='43'. That prefix → short_name mapping exists only implicitly, spread across three parsers. Nothing at build time, load time or ingest time asserts that a cited article exists.

Measured state (2026-07-22)​

63 distinct regulatory_basis strings across the registry. Resolved against the live corpus:

OutcomeCountShare
Resolves to a real lex_articles row4165%
Grammar cannot parse (no article, ranges, "et seq.")914%
Instrument absent from corpus or unmapped prefix813%
Article missing from an ingested corpus58%

35% of the legal bases shown to officers cannot be grounded, and the system renders all 63 identically.

This single fact explains the week's defects: the 15 mis-citations corrected on 2026-07-21 survived because nothing joined; FR-CMF citations pointed at a law absent from the corpus and failed silently; GwG s. 43 resolved to a bare heading because the row existed but held no law. It also explains a subtler one — the 2026-07-21 sweep corrected PEP to AMLR Art. 42-44, which is substantively right but is a range, a form no article lookup can satisfy. The correction was invisible to every machine in the chain.

Two further defects follow from the same seam: idx_lex_articles_number is a plain btree with no uniqueness on (regulation_id, article_number), and get_article() uses LIMIT 1 with no ORDER BY — so with duplicate rows it returns an arbitrary one. That is how "Wwft Art 3" returned a wrong heading rather than an error.

Decision​

Bind rules to the corpus through a structured, resolvable citation, and make resolution a build-time invariant rather than a render-time accident.

  1. One shared resolver. A single citation_resolver module is the sole source of truth for the citation grammar and the instrument-prefix → short_name mapping. RegPill, the copilot tool and citation_verifier consume it; none re-implements it.

  2. A tri-state resolution result, never a boolean: resolved (a real lex_articles row) · not_in_corpus (grammar understood, instrument or article absent) · unparseable (grammar did not understand the citation). The distinction is the point: "we have not ingested this law" and "this citation is malformed" are different facts and must not collapse.

  3. A contract test as a standing gate. Every regulatory_basis in the registry must either resolve, or appear in an explicit KNOWN_UNRESOLVED register with a reason and an owning issue. Adding a rule with an ungroundable citation fails CI. This converts the manual 2026-07-21 citation audit into an automatic invariant — the same "kill the class, not the instance" move as ADR-0118's corpus integrity gate.

  4. Fail-closed surfacing. A citation that cannot be grounded renders as "legal basis stated — article text not available for verification", never silently as a normal pill. This applies to citations the same honesty contract ADR-0067 applies to compliance outputs and ADR-0068 to country capability.

  5. Referential integrity at the database. A unique index on (regulation_id, article_number) plus deterministic ordering in get_article(), so a duplicate is a loud constraint violation at ingest rather than an arbitrary row at read.

The persisted regulatory_basis string is retained as the human-facing display form. This ADR does not migrate it to a tuple; it makes the string machine-checkable and adds the resolver as the single interpreter. A future ADR may promote the structured tuple to the persisted form once the resolver has proven the grammar against the full corpus.

Consequences​

Positive​

  • A citation that cannot be grounded is impossible to commit unnoticed.
  • The 22 currently-unresolvable citations become an explicit, reasoned register instead of silent failures — visible work rather than invisible debt.
  • One grammar in one place; the frontend regex families stop drifting from the backend lookup.
  • Officers see an honest "not verifiable" state instead of a pill that looks identical to a grounded one.
  • Duplicate article rows become a constraint violation at write time.

Negative​

  • The KNOWN_UNRESOLVED register is a hand-maintained list, and feedback_claim_vs_check warns that such lists go stale. Mitigated by requiring a reason + issue per entry and by asserting in the same test that every entry is still needed — an entry that now resolves fails the test, so the register cannot silently over-claim.
  • The unique index will fail to build if duplicates exist; the migration must detect and report them rather than silently dedupe.
  • Ranges (Art. 42-44) and instrument-only citations (Belgian AML Law) need a grammar decision rather than a lookup: a range resolves to its first article with the span recorded; an instrument-only citation resolves to the instrument, not an article, and is labelled as such.

Neutral​

  • No change to how rules fire, to severities, or to any risk computation. This ADR governs the legal basis label and its verifiability only.

Alternatives Considered​

Alternative 1: Persist a structured tuple now, drop the string​

Replace regulatory_basis: str with (jurisdiction, instrument_key, article, section). Rejected for sequencing, not merit: 17 of 63 citations do not parse under any current grammar, so the migration would have to invent structure for a quarter of the corpus before the grammar is proven. Build the resolver first, let it prove the grammar against real data, then promote the tuple. Recorded as the likely successor ADR.

Alternative 2: Validate at render time only (harden the three parsers)​

Keep the current architecture and make RegPill show a warning when a pill fails to deep-link. Rejected: the earliest a bad citation could be noticed is when an officer clicks it, the three parsers continue to drift, and it does nothing for the case-pack PDF where there is no click. It treats the symptom at the last possible moment.

Alternative 3: Do nothing — the citations are advisory labels​

Rejected on regulatory grounds. Under EU AI Act Art. 12 and AMLR every AI decision must be traceable to its stated basis; a legal basis that cannot be resolved to the text it names is not traceable. It also fails the project's cardinal doctrine — an ungroundable citation shown identically to a grounded one is a scrutiny-reducing output with no evidence trail.

Decision context​

  • Latency: resolution is a single indexed lookup, and the contract test runs at CI time, not request time. Render-time resolution is unchanged in shape. Not measured because no new per-request work is introduced.
  • Dependency surface: none. Pure-Python module over existing tables.
  • Debuggability: the tri-state result names which failure occurred, so a broken citation reports "instrument FR-CMF not in corpus" rather than an empty pill. The contract test failure names the offending rule.
  • Reversibility: hours. The resolver is additive; the unique index is a single migration; the fail-closed rendering is one component branch.
  • Blast radius: additive for the resolver and the test. Substitutive in two places — get_article() ordering and the RegPill unresolved state.
  • Alternative considered: persist the structured tuple immediately (Alternative 1) — rejected because the grammar is not yet proven against the 17 citations it cannot currently parse.

References​

  • Measured resolution rate and gap breakdown: this ADR, Context §Measured state
  • ADR-0118 (corpus integrity gate) — the same discovery-mechanism pattern
  • ADR-0067 (fail-closed compliance outputs), ADR-0068 (country capability) — the honesty contract this extends to citations
  • ADR-0016 (shared regulatory corpus) — the corpus half
  • Citation accuracy audit: docs/reviews/2026-07-21-citation-accuracy-audit-calibration-review.md
  • Issue #484 (corpus coverage), which the KNOWN_UNRESOLVED register references