Skip to main content

ADR-0113: Fail-Closed Per-Case Compliance Verdict

Date: 2026-07-17 Status: Accepted Deciders: Adrian (Soft4U BV), Claude (Fable 5) — from the 2026-07-17 live-UI compliance-officer assessment (epic #350)

Decision context:

  • Latency: verdict computed at API read time from one cases row + one agent_executions scan + the latest rule_evaluations row — three indexed queries inside the already-open tenant session, estimated < 15 ms p50 / < 60 ms p95 added to GET /api/cases/{id}/rule-evaluations; not measured under load because the endpoint is officer-interactive (single-case), not batch.
  • Dependency surface: zero new packages; one new backend module + one new API field; reuses the existing rule engine, coverage-state exclusion, and screening-evidence JSONB contracts.
  • Debuggability: the verdict carries ordered human-readable reasons and a per-escalator detail string; a computation failure degrades to an explicit insufficient_data verdict (logged), never a missing block — so a broken verdict is visibly amber, not silently green.
  • Reversibility: additive API field + one new module + template-data edits; revert = drop the verdict block and the template category constants (~2 h). The rule-engine tri-state is a new default-valued field on persisted JSON (old rows unaffected).
  • Blast radius: GET /rule-evaluations response (additive), RulesAppliedCard/ComplianceTab rendering, reasoning-template rule conditions (all templates), one new store call at end of investigation. The deterministic EBA risk spine (ADR-0020/0082/0089) is untouched — the verdict reads its outputs.
  • Alternative considered: fix only the frontend banner predicate (reject: leaves the persisted "0/10 triggered" lie and every other consumer of it intact; the defect is that the verdict is not a function of the evidence, not that one banner is green).

Context​

The 2026-07-17 live-UI assessment (epic #350) found the platform's single systemic failure: the per-case compliance verdict had come apart from the evidence the platform itself collected. The flagship OB Holding 1 OÜ case rendered "10 rules passed — All compliance rules passed — no red flags detected" while the same case held 14 CRITICAL entity_criminal_investigation findings (EPPO asset seizure, ~€31.6m money-laundering investigation), an outstanding sanctions review, a PEP hit, 0/5 documents, and — for the sibling case whose investigation was killed at the 30-minute activity timeout — no investigation results at all. Issues #351 (false clear), #353 (coverage chips render "covered" over 0% evidence), #354 (rules evaluate "passed" on missing inputs), #355 (advertised non-disableable escalators demonstrably did not fire) are the critical cluster.

Three stacked root causes were verified live:

  1. Dead rule taxonomy. Reasoning-template rules matched finding categories adverse_media_hit / sanctions_hit / pep_match — strings no agent emits. Agents emit adverse_media, adverse_social_media, sanctions_match, pep_review, entity_criminal_investigation. Every real case persisted rules_triggered=0 — the rules were dead code, the exact taxonomy-conflation class ADR-0094 documented for classify_material_type.
  2. Stale partial evaluation. The template was evaluated once, mid-pipeline, over only the registry/person/media phase outputs — before Tier-B adverse recall, synthesis, and the network phase append findings — and the persisted row was replayed verbatim by the API forever, including for workflows that later died.
  3. Fail-open rendering. rules_triggered === 0 rendered the green all-clear with no completeness state; {evaluated:false} (crash, store failure, or no template — three indistinguishable states) rendered as the benign "No reasoning template applied"; the Ontology chips were a hardcoded always-teal array with no data binding.

Additionally, the rule engine's AND-condition semantics evaluated each condition over the whole findings list independently, so a category condition could match finding A while a severity condition matched finding B — cross-finding conflation that made precise category+severity rules unwritable.

Decision​

Make the per-case compliance verdict a fail-closed function of the collected evidence, computed at read time from the case's live persisted state, with one shared token consumed by every per-case compliance surface.

  1. New pure verdict module backend/app/services/compliance_verdict.py: build_compliance_verdict(...) composes (a) the latest persisted rule evaluation, (b) the latest iteration of cases.additional_data->'investigation_results' — open findings (excluding coverage-state categories and verified severity), the deterministic sanctions_screening / screening_summary / material_check_incomplete / adverse_recall_coverage evidence, (c) resolved_requirements document state, (d) investigation/post-processing completeness (case-status set ∪ agent_executions terminality), and (e) fired regulatory escalators from additional_data.risk_assessment.auto_escalators. Output states: escalate > insufficient_data > clear — a hit on partial data is still a hit; missing inputs can only withhold a clear, never suppress an escalation. clear requires: complete investigation, all of sanctions/PEP/adverse-media assessed, all required documents validated, no critical/high open findings, no outstanding screening review, and an existing rule evaluation.
  2. Escalator trace per enforcement id (SANCTIONS_EXACT_MATCH, SANCTIONS_HIT, PEP_DETECTED, FATF_BLACKLIST, ENTITY_CRIMINAL_INVESTIGATION, NETWORK_SANCTIONS_CONFIRMED, PAYMENT_ACCOUNT_OUTSIDE_LICENCE[_CRITICAL], ADVERSE_CORPORATE_HISTORY, TAX_SOCIAL_DEBT): fired / clear / not_evaluated — <why>. The catalog is contract-tested against the enforcement sources (eba_risk_matrix.py, risk_matrix_service.py) so it cannot drift into a parallel vocabulary — the settings-page display list (PEP_UBO …) is explicitly not the source.
  3. Rule-engine honesty: new ConditionType.CATEGORY_SEVERITY_GTE requiring category ∈ set AND severity ≥ threshold on the same finding (and never verified); per-rule status: triggered | passed | not_evaluated where finding-scanning rules over an empty findings list report not_evaluated, not passed; ReasoningResult.rules_not_evaluated.
  4. Template taxonomy fix: shared category constants (SANCTIONS_HIT_CATEGORIES/PEP_HIT_CATEGORIES/ADVERSE_MEDIA_CATEGORIES/ CRIMINAL_CATEGORIES) matching what agents actually emit, applied across all templates; new eu_generic_criminal_investigation CRITICAL rule (confidence cap 15, mandatory EDD).
  5. Final re-evaluation: at the end of the investigation (after Tier-B recall/network findings are final) the template is re-evaluated over the full finding set and stored, so the persisted row reflects the complete evidence; the early mid-pipeline evaluation is kept for the synthesis prompt.
  6. API: GET /api/cases/{workflow_id}/rule-evaluations gains an additive verdict block (all legacy fields preserved). A verdict-computation exception degrades to an explicit insufficient_data verdict — never a missing block, never green.
  7. Frontend: RulesAppliedCard renders the three states (red escalation with the finding count inline; amber "verdict withheld — insufficient inputs (fail-closed)" with the reasons; green only on clear, and the "no red flags detected" sentence only when zero open findings of any severity). A missing verdict block renders the amber fail-closed default. The Ontology "Screening"/"Documents" chips bind to the verdict coverage; the AMLR-Sanctions/PEP standards chips prefer the verdict coverage over the dead graph tags.

Adversarial-verification hardening (same PR)​

Five independent refuters were run against the first implementation; all five found real defects, which are fixed and regression-pinned in the same change:

  1. Coverage-state detection is EXACT-match, not suffix-match. The first cut excluded any category ending _gap/_not_assessed/_coverage/_recall_gap from open findings — but Finding.category is free-form LLM text, so a real HIGH finding named internal_controls_gap was excluded and produced a green "no red flags" (the ADR-0094 defect class, inverted). is_coverage_state_category now matches an exact canonical frozenset of the deterministic pipeline's coverage markers only.
  2. Material-findings semantics change + restored override floor (ADR-0089/0100 interplay). Routing classify_material_type through the coverage-state exclusion stops a stale sanctions_screening_gap marker being persisted/re-injected as an established material sanctions finding (correct: a "screen did not run" marker floored the entity at HIGH forever, even after clean re-screens) — but it also silently removed the ADR-0100 override floor that marker had provided. Restored case-scoped: RiskOverrideService now floors a downward override at HIGH while the case's latest run carries a live coverage-gap marker (sanctions_screening_gap / adverse_media_not_assessed / screening_error) — current-run-scoped, so the stale entity-floor defect cannot recur.
  3. Vacuous coverage killed. PEP coverage requires ≥1 person actually screened (an all-zero screening_summary from an empty director roster is not_assessed, not vacuously "assessed"); sanction_hit/needs_review badges are escalation signals; malformed evidence sub-shapes and unrecognized finding severities each add a fail-closed gap reason.
  4. Missing risk assessment fails closed. A case whose initial EBA assessment was guard-and-swallowed (no risk_assessment on the case) can no longer show a clear verdict or "clear" escalator-trace rows — every non-fired row renders not_evaluated — deterministic risk assessment never computed, and a deterministic PEP hit without a propagated PEP_DETECTED signal renders not_evaluated, never clear.
  5. Final re-evaluation parity + monotonic merge. The final row applies the same NBB financial-bridge evidence as the early evaluation (no fabricated SOURCE_MISSING("nbb") triggers), and is merged monotonically with the early row (a rule triggered on early evidence never silently un-triggers because synthesis renamed a category; confidence_cap = min of the two) before persisting.

Known follow-up (tracked, not in this change): deterministic PEP hits from run_verification_checks do not propagate into the reassess pep_detected signal (checkpoint-2 signal derivation reads only HIGH/CRITICAL LLM findings) — the verdict now surfaces this as not_evaluated rather than hiding it; the signal-propagation fix touches workflow code and ships separately.

Consequences​

Positive​

  • The flagship contradiction is structurally impossible: no green surface can render while the same case holds unresolved ≥high findings, outstanding screening review, unassessed coverage, or an incomplete investigation.
  • A killed/failed workflow shows an explicit amber "not assessed — do not rely" state instead of replaying its pre-crash "passed" row (issue #354's exact scenario).
  • The escalator trace makes a non-firing floor visible per case (issue #355's ask), keyed to the enforcement vocabulary with a drift-blocking contract test.
  • The rules can actually fire: the rejected OB fixture now triggers criminal (CRITICAL), adverse-media (HIGH) and PEP rules instead of "10 rules passed".

Negative​

  • Read-time composition adds queries to the rule-evaluations endpoint (~ms; officer-interactive).
  • insufficient_data is now the common state for every case before REVIEW_PENDING — officers lose the (false) comfort of an early green; this is the intended cost of honesty.
  • Historical rule_evaluations rows remain as written (0-triggered); the live verdict block supersedes their presentation but the audit history shows the old dead-taxonomy outputs.
  • Document coverage is fail-closed to 0 validated when validation results are not persisted on the case row — decided legacy cases may show insufficient_data on this axis (honest: their documents genuinely were never validated, see #380).

Neutral​

  • The Standards portfolio page (Neo4j standards_refs) still under-reports screening coverage because the Finding-tag writer is dead code — tracked separately (#364/#353 follow-up); the contradiction with the case surface disappears because the case side no longer claims green.
  • Entity-keyed disposition (a prior Reject blocking sibling cases) is Workstream B (#352) and composes on top of this verdict token later.

Alternatives Considered​

Alternative 1: Frontend-only fix (change the banner predicate)​

  • Render amber unless the case status is terminal, keep everything else.
  • Why rejected: leaves the persisted "0/10 triggered" false record, the dead rule taxonomy, and every other consumer (synthesis context, EVOI beliefs, intelligence trends) reading a verdict that is not a function of the evidence. The defect is data-layer, not presentation.

Alternative 2: Move the whole verdict into the Temporal workflow (persist a verdict row per state change)​

  • Compute and persist the verdict as workflow state at every checkpoint.
  • Why rejected: the verdict must reflect evidence that changes outside workflow checkpoints (killed workflows, later monitoring evidence) — exactly the staleness class being fixed; read-time composition over persisted state is deterministic, cheaper, and cannot go stale.

Alternative 3: Re-derive rules live on every GET and drop persistence​

  • Why rejected: the persisted row is the audit trail (EU AI Act Art. 12) of what the pipeline evaluated when; three existing read sites already re-derive with different inputs and disagree — the fix consolidates on persisted-evaluation + live-verdict-composition, with the final full-evidence re-evaluation closing the staleness gap at the source.