Skip to main content

ADR-0165: PEPPOL Verify's no-LLM claim is withdrawn, and a withholding check that did not run is a visible FLAG

Date: 2026-08-01 Status: Accepted Supersedes: ADR-0006 Deciders: Adrian (Soft4U BV), Claude Opus 5

Supersedes: ADR-0006 (PEPPOL Verify as Synchronous REST API)

Decision context:
- Latency: unchanged. The Rule 3c change is a constructor call on a branch that
  previously returned early; the rest of this ADR is a records correction with no
  runtime path. The PEPPOL latency figure is separately contested and deliberately
  NOT re-asserted here (issue #949).
- Dependency surface: none added. No new package, no new service, no new config key.
- Debuggability: improves. A withholding check that produced nothing now reaches the
  officer as `INHOUDINGSPLICHT_UNAVAILABLE` from every caller rather than from only
  the callers that happened to build a check object.
- Reversibility: minutes. The guard is six lines in one function
  (`risk_engine.py:101-106`); the document changes are text.
- Blast radius: additive. One new flag can appear on a verification that previously
  carried none; no existing flag's type, severity or wording changes (pinned by
  `test_risk_engine_withholding_gap_945.py:146-148`).
- Alternative considered: put the guard in the PEPPOL bridge
  (`peppol_verification_service.py:169-170`). Rejected — it fixes one caller, and the
  invariant has to hold for every caller (§ "Why the engine, not the bridge").

Context​

ADR-0006 decided how PEPPOL Verify is deployed (a synchronous REST API on the existing FastAPI app, not a Temporal workflow and not a microservice). That part of the decision is sound and still in force. This ADR exists because of a different sentence in the same document.

ADR-0006 states, as part of its rationale:

No LLM involvement:

  • The entire pipeline is deterministic: API lookups, string matching (Jaro-Winkler), rule evaluation, and structured output
  • Zero inference cost, zero hallucination risk — every data point is cited from an authoritative source

That is false as shipped, and it has been false since the withholding (inhoudingsplicht) check joined the verify path — a fourth source ADR-0006 never named. ADR-0006 was written on 2026-02-20 against a three-source pipeline; it was never revisited when the fourth arrived.

The evidence. PeppolVerificationService.verify gathers four branches, not three (peppol_verification_service.py:115-119). The fourth is the withholding check, and its determination is produced by a language model:

  • inhoudingsplicht_service.py:697-699 constructs a pydantic-ai Agent[None, _BrightDataExtraction] whose structured output is the determination.
  • The model is settings.belgian_scraping_agent_model (:692) = openai:gpt-4.1-mini (config.py:794).
  • The model drives the fetch: Bright Data's Web Unlocker is attached as an MCP toolset and the system prompt directs the agent to call scrape_as_markdown on https://www.checkinhoudingsplicht.be/result/{en_short} (:688), so the page enters the model's context as a tool result and leaves it as a debt_free_social / debt_free_tax pair.
  • Attaching a tool is not requiring it (added on review): toolsets=[server] makes scrape_as_markdown available, and pydantic-ai accepts a _BrightDataExtraction the model emits having called nothing — in which case no page ever entered its context and both booleans are invention. As first written this record asserted the call as fact while the code accepted run_result.output unconditionally, which is the claim-vs-check pattern this ADR exists to correct, reproduced inside the correction. _brightdata_scrape now REJECTS an output carrying no scrape_as_markdown tool return (_scrape_tool_returned), routing it into the un-assessed branch a Bright Data outage already produces (debt_free_* = None, unavailable=True, GREY, no certificate). Accepted evidence is narrow — a ToolReturnPart for that tool; a ToolCallPart, a RetryPromptPart, another tool's return and an unrecognised result shape all read as not fetched (fail-closed, ADR-0067). CACHE_NAMESPACE bumped v2->v3: entries minted before the gate carry no record of whether the page was read, and check_single serves a cache hit ahead of the gap check. The gate establishes that a page was fetched — it does not establish that it was the right page or that the model read it correctly, which is item 5's territory, still open.
  • There is no deterministic parse of that page anywhere on the path. The model is the sole producer of those two booleans.
  • The fallback path (belgian_scraping_agent.py:416-432) constructs a second Agent, resolved through the tier table to mid — more expensive than the primary path, not less.

Those two booleans are the inputs to SOCIAL_DEBT_DETECTED and TAX_DEBT_DETECTED (risk_engine.py:135-148) — the only two FAIL-severity flags in the whole PEPPOL engine whose truth value originates in a generative model. A FAIL is "do not onboard without further investigation". So the claim is not merely stale trivia: it is wrong at exactly the point where it matters most to a reader deciding whether to trust the service.

The claim is not confined to ADR-0006. It has four renditions, and the ones a non-employee reads are the strongest:

WhereLineWhat it says
docs/adr/ADR-0006-peppol-verify-as-rest-api.md25, 38-40the canonical record
docusaurus/trust-relay/docs/adr/0006-peppol-rest.md41, 43, 50the published rendition — restates it three times, including a "Positive consequence" bullet the canonical does not have
docusaurus/trust-relay/docs/api/webhooks.md46"The entire pipeline is deterministic -- no LLM involvement" — directly under a numbered source list whose item 4 is the withholding check
docusaurus/trust-relay/docs/architecture/cost-monitoring.md117the claim in operational form: PEPPOL is listed as a "non-LLM wrapper" with "no token tracking since they don't invoke PydanticAI agents"

The last one is the most consequential, because it is not just a description — it is the reason the inference cost is unmeasured. agent_executions.cost_eur is populated only when a caller passes a model and token counts into update_status (cost-monitoring.md:108-118 documents the five-step bridge). The withholding call is outside that bridge by design, on the strength of a classification that is wrong. A document asserting a cost is zero, and a pipeline not instrumented to measure it, are the same mistake wearing two hats.

The second defect, found while correcting the first. risk_engine.evaluate took inhoudingsplicht: InhoudingsplichtCheck | None = None and guarded all three withholding rules on the object being present. inhoudingsplicht is None means the caller produced no check object at all — the source failed above the boundary, or the branch that builds the object was removed. Epistemically that is identical to a check that ran and returned no verdict: the determination did not happen. But only the second shape reached Rule 3c.

This was a latent shape, not a live false clear, and the distinction is load-bearing: every documented failure today is caught below that boundary and yields a tri-state None inside a check object that still gets built, so Rule 3c does fire on the live path. The hole is what happens when someone severs the LLM branch — the exact change issue #945 contemplates. Removing the branch would leave inhoudingsplicht_check = None (peppol_verification_service.py:169-170), which under the old guard produced no flag at all. Fixing the model without fixing the guard makes the failure mode worse, not better.

Decision​

1. Withdraw the no-LLM claim in all four renditions, and supersede ADR-0006.

ADR-0006's status becomes Superseded by ADR-0165; its body is left untouched, per the supersession discipline — the historical record of what was decided on 2026-02-20 is not rewritten. The deployment decisions it made (REST not Temporal, mounted on the existing FastAPI app, API-key auth, in-memory rate limiting) are carried forward by this ADR unchanged. What is not carried forward is the "No LLM involvement" rationale.

The published rendition (0006-peppol-rest.md) is treated asymmetrically and deliberately: it gets the status flip and a correction banner at the top of the page. A header field reading Superseded is adequate notice in an engineering register that a reader arrives at deliberately; it is not adequate notice on a page a prospect lands on from a search result, three screens above the sentence that misleads them. The banner does not alter the body. This asymmetry is recorded here so it is a decision rather than an inconsistency, and because the mirror generator (docusaurus/trust-relay/scripts/generate-adr-mirror.js) is generate-if-missing and never overwrites — the published page will not self-heal from a canonical fix and must be maintained by hand.

webhooks.md:46 and cost-monitoring.md:117 are not ADRs and are corrected in place with a dated note, in the same style as the PRD.

2. State the cost honestly: not measured, not zero.

The inference line is recorded as not measured. The model and its token prices are known (openai/gpt-4.1-mini at €0.28/€1.10 per 1M tokens; openai/gpt-5.2 at €2.30/€9.20 — Alembic 046_cost_monitoring.py:76-77, source='manual', effective 2026-04-01, internal estimates and not vendor invoices). The token volume of a single extraction is not: the model is fed a whole scraped page whose length nothing in this repository measures, and the share of verifications falling through to the more expensive fallback is not instrumented either.

Readers are not pointed at agent_executions.cost_eur for this number, because that column does not contain it — see the instrumentation gap above. Pointing at telemetry that was never wired would replace a false zero with a false promise. The correct statement is that the line is unmeasured and that measuring it requires instrumenting the call first; that work is tracked, not claimed.

3. Rule 3c fires when the withholding source wholly fails.

risk_engine.evaluate normalises an absent check object into an explicitly un-assessed InhoudingsplichtCheck before any rule runs (risk_engine.py:101-106). Both shapes — "ran, no verdict" and "no check object at all" — then travel the same path, so Rule 3c fires for both, for every caller.

Why the engine, not the bridge​

The obvious place for this guard is the PEPPOL bridge, where the None is produced (peppol_verification_service.py:169-170). That was rejected. The bridge stops constructing the object on exactly one failure path; the invariant that must hold is "a withholding check that did not run is a visible FLAG, never a silent clean pass", and that is a property of the engine's contract, not of one caller's error handling. Placing it in the engine means a future caller — a second bridge, a batch re-screen, a test harness — inherits the invariant instead of having to remember it. It also means a later rule cannot reintroduce the hole, because None no longer exists downstream of line 106.

What this ADR does not decide​

  • Whether the model stays. Issue #945 item 5 asks whether a generative model may sit anywhere on a path producing a FAIL-severity regulatory flag. That decision is blocked on a factual question this repository cannot answer — whether RSZ/ONSS operate a licensable Art. 30bis/30ter consultation channel. This ADR records that the model is there and bounds what its failure can claim. It does not bless it.
  • The sub-2-second latency claim. ADR-0006 asserts it twice and the published rendition four times; it is contested by issue #949 with its own evidence. It is deliberately not re-asserted in this ADR. Superseding ADR-0006 must not become the mechanism by which a second unverified claim is laundered into a fresh Accepted document.
  • The VIES-unavailable flag variant. risk_engine.py:171-176 emits VAT_INVALID at FLAG severity when VIES is unavailable, reusing the same flag type as a confirmed invalid VAT at FAIL. So "the VAT is invalid" and "we could not check the VAT" are distinguishable by severity and description but not by type — precisely the conflation that INHOUDINGSPLICHT_UNAVAILABLE exists to avoid on the withholding side (packages/trustrelay-models/src/trustrelay_models/peppol.py:36-39). The behaviour is documented in the PRD by this PR and left unchanged in code: giving the unavailable case its own type is an API-contract change for existing consumers and belongs in the PEPPOL extraction work, not in a documentation correction.

Consequences​

Positive​

  • The document a commercial counterparty reads no longer claims a property the service does not have. That is the whole point: an overclaim in a PRD or a published ADR is the same defect class as a false clear in the product.
  • The failure mode is bounded structurally rather than by convention. An extraction that determines nothing yields INHOUDINGSPLICHT_UNAVAILABLE from every caller, so severing the LLM branch — the change #945 contemplates — can no longer silently turn every failed check into no flag at all.
  • The instrumentation gap is now named in the place that caused it (cost-monitoring.md), so the next person to ask "what does this cost?" is told the truth instead of being sent to an empty column.

Negative​

  • The €2.00-per-verification economics can no longer be stated as a single number. Every margin figure becomes an upper bound with an unmeasured line beneath it, which is weaker commercial material than the 97% it replaces. It is also true, and the previous figure was arrived at by booking a real cost at zero.
  • A verification that previously carried no flag can now carry INHOUDINGSPLICHT_UNAVAILABLE, which costs the caller a manual review. This is the intended direction (scrutiny added, never suppressed) but it is a real cost to a customer whose withholding source was merely flaky.
  • Superseding ADR-0006 means a reader must now follow one hop to get the current deployment decision. The alternative — editing ADR-0006 in place — would have destroyed the record of what was actually decided in February, which is worse.
  • Two claims in ADR-0006 are now in different states: one withdrawn here, one contested elsewhere (#949). Until #949 lands, the honest reading of the latency assertion is "unverified", and nothing in this ADR makes that visible on the published page beyond the supersession banner.

Neutral​

  • No migration, no configuration flag, no new dependency. The guard is unconditional because a fail-closed default needs no rollout ramp — there is no state in which emitting the gap flag is the wrong answer.
  • The evidence hash, response schema and API contract are unchanged; consumers see at most one additional flag object of an already-published type.

Alternatives Considered​

Alternative 1: Correct the PRD only, leave ADR-0006 Accepted​

  • Fix the PRD (which this PR does) and leave the ADR register alone.
  • Why rejected: it produces two live documents that contradict each other, one of which is the architectural record of record and remains Accepted. An engineer reading docs/adr/ would find a binding decision asserting no LLM; an engineer reading the PRD would find the opposite. The published rendition and webhooks.md would keep the claim in front of customers regardless of what the PRD says. The finding that prompted this work names precisely this outcome as the failure.

Alternative 2: Edit ADR-0006 in place — strike the "No LLM involvement" section​

  • Delete or rewrite the offending rationale inside the existing Accepted ADR.
  • Why rejected: it destroys the record. ADR bodies are immutable once Accepted (s4u-adr §10); the decision on 2026-02-20 genuinely was made partly on a no-LLM premise, and erasing that erases why the service was scoped as it was. ADR-0122 is the in-repo precedent for the opposite move — retiring something by writing a new ADR that cites it. Silently editing an accepted record to match current reality is the documentation form of overwriting an audit row.

Alternative 3: Give the guard to the PEPPOL bridge instead of the engine​

  • Construct the un-assessed check object in peppol_verification_service where the None originates.
  • Why rejected: covered above — it binds one caller, and the invariant is a property of the engine's contract. It also leaves None alive inside evaluate, so any rule added later can reintroduce the hole.

Alternative 4: Estimate the inference cost rather than record it as unmeasured​

  • Pick a plausible token count per page, multiply by the known per-token prices, publish a number.
  • Why rejected: it would replace an unsourced zero with an unsourced non-zero, in a document whose whole correction is that its numbers were not sourced. The page length is not measured and the fallback-path share is not instrumented, so any figure would be invented. "Not measured" is the honest state and is what the reader needs in order to know the number is missing rather than small.