Skip to main content

ADR-0111: Finding Correctness Re-Verifier

Date: 2026-07-13 Status: Accepted Deciders: Adrian (Soft4U), Claude Opus 4.8

Context​

The investigation quality gate shipped in #302 part 1 (quality_scorer.py + quality_gate.quality_gate_forces_review) grades the structural quality of an OSINT investigation — coverage, source count, corroboration shape — and can force a case to officer review when the investigation looks thin. It does not ask whether the individual decision-driving findings the report asserts are actually true. A report can be structurally rich and internally consistent while carrying a finding that the live sources no longer support — a sanctions "hit" that the OpenSanctions record does not contain, a criminal-prosecution claim that no adverse-media source corroborates. The most catastrophic compliance failure is not a thin investigation; it is a confident, well-formed report that is wrong about a material fact, in either direction (a fabricated hit that rejects a clean customer, or — worse — a claim that looks resolved but was never re-checked).

The forces at play:

  • Presence ≠ evidence (the standing 2026-06-26 OB Holding defect class): a "no matches" re-check record is not a confirmation, and an off-topic adverse result does not corroborate a specific criminal claim. Any re-verifier must encode this or it manufactures false confidence.
  • Add scrutiny, never suppress (MEMORY regulatory principle, EU AI Act Art. 14 / AMLR): the only safe output of a correctness check is more officer oversight. It must never lower a tier, drop a finding, or blend a wrong finding into an averaged score where it can be diluted below the review threshold.
  • Fail-closed / fail-toward-scrutiny (ADR-0067): a re-check that cannot run (timeout, provider error, unmapped material type) must resolve to a verdict that raises scrutiny, never a silent pass.
  • Auditability (ADR-0064, EU AI Act Art. 12): a verdict that can force a compliance decision must leave an immutable, retrievable trail.
  • Determinism (Calibration Review): the gate outcome for the same live evidence must be stable — an LLM leg, where used, runs at temperature 0 and is reduced to a boolean by rules.

The re-check machinery already exists and is trusted: run_kyc_screening (the real OpenSanctions person screener, ADR-0101/0103), AdverseMediaService (the adverse-media agent adapter), and material_findings.classify_material_type (the single-source material-type classifier). The open question is how to compose them into a correctness check without contaminating the existing structural gate or the sanctions-screening evidence store.

Decision​

Ship a new, independent finding_correctness_verifier service that re-derives decision-driving findings from live sources and classifies each one confirmed | contradicted | unsupported | indeterminate. Any non-confirmed decision-driving finding hard-forces needs_review — it is never averaged into a blended score. The verifier is composed into the existing score_investigation_quality_activity (it is not part of quality_scorer.py's structural scoring), and #302 part 1's quality_gate_forces_review is extended to read its correctness_needs_review flag (honored only when the gate is enforcing, i.e. advisory_only=False).

Concretely:

  • Scope — decision-driving adverse findings only. select_decision_driving_findings keeps high/critical-severity findings, plus material-typed findings (sanctions, criminal, enforcement, regulatory_action, adverse_media, …) that assert an adverse condition. Category marks the topic, not the verdict: OSINT emits clean facts ("No matches found") under the SAME material category at severity low, so a material finding in the clean/informational severity band (low/info/none) is excluded — re-verifying a clean fact would re-derive a clear screen and mislabel it contradicted, a false correctness failure on a clean report. Coverage-state markers (e.g. adverse_media_recall_gap) are excluded — classify_material_type already returns None.
  • Live re-derivation, entity-type-aware. rederive_finding dispatches per material type and per entity_type: sanctions → the natural-person KYC path run_kyc_screening for entity_type="person", else (KYB default) the company OpenSanctions screen screen_opensanctions(entity_type="company") — a company must not be re-derived against the person schema; adverse/criminal/enforcement/regulatory → AdverseMediaService().search. Bounded by finding_correctness_recheck_timeout_seconds (default 20s) and finding_correctness_max_rechecks (default 6). A timeout, error, or unmapped type returns None.
  • Presence ≠ evidence classification. classify_finding_correctness maps a live re-check to a verdict against each re-check's REAL return contract: None → indeterminate; sanctions → the real run_kyc_screening status is authoritative (clear → contradicted; hit/warning → confirmed, never suppressing a real screening signal; error/missing → indeterminate); adverse types → the real AdverseMediaService.search {"hits": int, "summary": str} contract: hits <= 0 → unsupported, and hits > 0 is confirmed only if the temperature-0 _fuzzy_match_adverse leg (LLM reduced to a boolean by rules, settings.quality_scoring_model) says the summary actually corroborates this claim. An error in the fuzzy leg returns False (fail-toward-scrutiny).
  • Hard trigger, not blended. correctness_verdict sets correctness_needs_review = any(verdict is not CONFIRMED). Nothing here lowers a tier or removes a finding.
  • Persistence to the immutable audit trail, NOT ScreeningResultRecord. Each verdict is written as a finding_correctness_verified event via AuditService.log_event (append-only audit_events, ADR-0064; per-tenant hash chain, ADR-0109), guard-and- swallow so a log hiccup never fails the verifier. The verdicts also ride on result["quality_gate"]["correctness_verdicts"] (retrievable with the report). Deliberately not the ScreeningResultRecord table: that table is sanctions- screening evidence that the risk engine and monitoring loop read; a correctness verdict mislabelled as a screening row would contaminate a compliance signal (a contradicted verdict is a statement about a finding, not a fresh screen result). The re-checks' own screening rows (written by run_kyc_screening itself) are unaffected.
  • Dark-launched. finding_correctness_verification_enabled defaults False; the flag flip is Calibration-Review-gated and out of scope for this ADR. With the flag off, score_investigation_quality_activity output carries no correctness keys — behaviour is unchanged.
  • Fail-closed wiring. In the activity, any verifier exception → correctness_needs_review = True (never a silent pass).

Consequences​

Positive​

  • A fabricated or stale material finding can no longer pass silently: re-derivation from live sources catches it, and a non-confirmed decision-driving finding hard-forces a second human look. This closes the "confident report, wrong material fact" gap that structural scoring cannot see.
  • Every verdict lands on the immutable, tamper-evident audit_events trail (ADR-0064/0109) and on the retrievable report payload — full EU AI Act Art. 12 traceability for a check that can force a compliance decision.
  • Reuses trusted, single-source machinery (run_kyc_screening, AdverseMediaService, classify_material_type, #302-p1's quality_gate_forces_review) rather than re-implementing screening or classification.
  • Strictly additive to scrutiny: it can only add an officer, never suppress a signal or move a tier — safe by construction.

Negative​

  • Live-source dependency when enabled. With the flag on, scoring re-invokes OpenSanctions and the adverse-media provider (up to 6 re-checks × 20s), adding latency and real API spend to the scoring activity and coupling the gate to provider availability. Fail-toward-scrutiny means an outage floods needs_review rather than clearing cases — safe, but noisy.
  • Evidence-contract seams found AND fixed during build. The mandatory real-entity validation (Task 9) surfaced two shape mismatches between the live re-checks and the classifier — the happy-path unit fixtures had encoded fictional contracts ({"sanctions_hit": bool}, a results list) that no real caller produces, so they were green-but-wrong. Both were fixed in this change: the classifier now reads the real run_kyc_screening status field and the real AdverseMediaService.search {"hits": int, "summary": str} contract, the unit fixtures were corrected to the real shapes, and test_genuine_sanctions_hit_is_confirmed pins that confirmed is reachable through the real path (the case the bug made impossible). The seams are recorded here because they are the cautionary lesson: content-grading fixtures MUST mirror real provider contracts, and only a real-entity validation catches the divergence.
  • LLM leg for fuzzy adverse matching. The non-empty adverse branch runs a temperature-0 LLM call; determinism depends on the provider honoring temperature 0, and it adds a model call per non-empty adverse re-check.

Neutral​

  • No database migration: the verdict rides on the existing quality_gate result blob and the existing audit_events table.
  • The verifier lives beside, not inside, quality_scorer.py; the structural score is untouched. A future consolidation could unify their orchestration, but the separation is intentional (structural quality vs. finding correctness are different questions).

Alternatives Considered​

Alternative 1: Fold correctness into quality_scorer.py's blended structural score​

  • Extend the existing structural scorer to also weigh finding correctness and let the single composite score drive the gate.
  • Why rejected: a contradicted decision-driving finding is a hard safety condition, not a quality gradient. Averaging it into a composite lets a high structural score dilute a wrong sanctions finding below the review threshold — the exact suppression the "add scrutiny, never suppress" principle forbids. The correctness signal must be a hard boolean trigger, orthogonal to the structural score.

Alternative 2: Persist verdicts as ScreeningResultRecord rows​

  • Reuse the existing sanctions-screening evidence table to store each correctness verdict, getting RLS and append-only semantics "for free."
  • Why rejected: that table is the sanctions-screening evidence the risk engine and monitoring loop read to derive live signals. A correctness verdict (a statement about a finding, e.g. contradicted) is not a screen result; writing it there would inject a synthetic row into a compliance signal source and risk mislabelling it as evidence. The append-only audit_events trail (ADR-0064/0109) is the correct home for a decision record, and it is already immutable, per-tenant hash-chained, and retrievable.

Decision context​

  • Latency: Dark-launched (default off) → 0 ms today. When enabled: up to finding_correctness_max_rechecks (6) live re-checks per investigation, each bounded by finding_correctness_recheck_timeout_seconds (20s), inside the scoring activity — off the officer's synchronous request path. Not measured against live providers because the flag flip (and its Calibration Review) is out of scope for this ADR.
  • Dependency surface: No new packages. Reuses run_kyc_screening (OpenSanctions), AdverseMediaService (Tavily/BrightData), pydantic_ai.Agent (already a dependency, temperature 0), AuditService, and classify_material_type. Owned lines: one ~230-line service + one gate-predicate clause + one activity insertion block.
  • Debuggability: Each verdict is an immutable finding_correctness_verified audit_events row and rides on quality_gate.correctness_verdicts in the report. Guard-and-swallow paths log warning with the case id and material type; the 4-way verdict enum makes the failure mode legible ("indeterminate" = re-check could not run).
  • Reversibility: A single config flip (finding_correctness_verification_enabled = False) fully disables the feature with no residue; no migration to unwind. Undo cost: minutes.
  • Blast radius: Additive and substitution-free. Adds correctness_needs_review / correctness_verdicts keys to the quality_gate blob and one OR-term to quality_gate_forces_review; touches one activity and one predicate. It never lowers a tier, drops a finding, or writes to a signal-bearing table.
  • Alternative considered: Blend correctness into the structural composite score — rejected because a contradicted material finding must hard-force review, not be averaged below the threshold.

Revision notes​

2026-07-18 — Concurrent rechecks, per-type timeout, clean/network exclusion (issues #424, #425)​

The second live validation run (wf_1b76902d78e4, on the fully-fixed post-#420/#422 code) proved the verifier now runs to completion and does REAL re-derivation — and surfaced three live-latency / classification defects. The original Decision above is unchanged; this note records the calibration fixes (all strictly additive to scrutiny, never suppressing a signal).

  • #424 — 20 s per-recheck timeout too short for live adverse media → all-INDETERMINATE. The live multi-provider adverse-media search (Tavily/BrightData) routinely exceeds 20 s, so every adverse/criminal re-derive timed out → indeterminate; the run's verdicts were {contradicted: 2, indeterminate: 4, confirmed: 0} — the verifier could never CONFIRM a genuinely-true finding (the real EPPO criminal finding included). Sanctions re-derives (the fast OpenSanctions screen) DID complete — the squeeze was adverse-specific. Root cause of the wall-clock squeeze: correctness_verdict ran the rechecks sequentially, so N × timeout stacked against the 180 s hosting-activity deadline and each recheck had to stay tiny.

    • Fix (primary): the rechecks now run concurrently via asyncio.gather (each per-finding rederive → classify chain is one gathered coroutine). Total wall-clock ≈ one recheck (~45 s), well inside the 180 s ceiling. gather preserves input order, so the verdicts list and the audit_events rows written from it stay deterministic.
    • Fix (budget): a new per-material-type timeout — adverse/criminal use finding_correctness_adverse_recheck_timeout_seconds (default 45 s); sanctions and any other type keep the fast finding_correctness_recheck_timeout_seconds (~20 s). Each is still capped to the remaining activity deadline, and the deadline floor-skip is preserved.
    • Alternative (noted, not taken): re-derive adverse findings from the within-case adverse-media result the investigation already produced this iteration (a cache) — faster and still evidence-based, but slightly less independent. Concurrency + a longer timeout was preferred to keep the re-derive a genuinely independent second look; the cache remains a future latency optimisation.

  • #425 — clean / network-scoped findings falsely CONTRADICTED. select_decision_driving_findings now excludes from correctness verification (before either selection branch) two shapes a subject re-derive cannot honestly test — REMOVING a false contradiction, never suppressing a signal (the underlying finding still stands; escalators/verdict unaffected):

    • (a) Clean screening FACTS ("EU sanctions screening: no designation evidenced …") assert the OPPOSITE of a hit, so a clear re-derive CONFIRMS them — yet the classifier maps sanctions clear → contradicted. Why the old _CLEAN_SEVERITIES exclusion missed it: a CONFIRMED-CLEAN result carries severity verified, NOT low — outside the old {low, info, informational, none} set — so the material branch selected it and the clear re-derive mislabelled it. Fix: verified/verified_clear added to _CLEAN_SEVERITIES (aligning with compliance_verdict), AND a robust _is_clean_screening_finding detector on the finding's OWN clear signal (hit=False/status="clear") plus clear-result title/description semantics — not severity alone. A genuine high/critical sanctions HIT is still selected.
    • (b) Network-scoped findings (type network_sanctions_review_required, "Network entities flagged …") make a claim about RELATED entities, not the subject; the re-derive screened the SUBJECT, got clear, → contradicted (a category error). Fix: _is_network_scoped_finding excludes findings whose taxonomy/narrative scopes the claim to the network. (Scoping the re-derive to each network entity is possible but out of scope; exclusion is the simpler, correct fix.)

  • Config: added finding_correctness_adverse_recheck_timeout_seconds: int = 45.

  • No migration, no contract change; feature stays dark-launched (finding_correctness_verification_enabled=False, Calibration-Review-gated).

Revision — 2026-07-18 (#424, live re-validation wf_8f1e1c91359e): dedicated activity + serial rechecks​

The prior revision's concurrent-rechecks-inside-the-scorer fix failed on the next live run: the LLM scorer ate ~140s of the shared 180s first, and the 6 concurrent rechecks self-contended on the process-global 3-slot BrightData semaphore (18–22s slot-waits inside each recheck's timeout) → every adverse re-derive timed out to INDETERMINATE (the EPPO criminal target included). Fix: a dedicated verify_finding_correctness Temporal activity (own 300s budget, called after score_investigation_quality at both score sites, correctness_* merged into quality_gate) running rechecks serially, highest-severity first — one BrightData call in flight → ~no slot-wait, criminal/sanctions findings re-derived first. No physical 4th BrightData slot (would breach the ADR-0077/0095 ceiling); serial-in-dedicated-activity delivers the reserved-lane outcome without it. Dark-launch invariant preserved (activity returns {} when off); fail-closed at both activity and workflow layers. Workflow-code change → terminate stale workflows on redeploy. #428 (gambling-stem casino anchor) is in ADR-0094.

Revision — 2026-07-18 (live re-validation wf_f87cd3ea2fc4): corroboration-window stability​

The dedicated-activity fix worked (verifier completes, no timeouts), but the EPPO criminal finding graded unsupported on one run and confirmed on a direct re-derive of the same finding — a stability defect. Root cause: the corroboration summary is built from only the top-5 adverse findings, and the search ranking is non-deterministic, so a corroborating item ranked outside the top 5 was missed. Fix: widen the verifier's window — AdverseMediaService.search(summary_findings=N) (default 5 for the scan pipeline; the re-derive passes finding_correctness_corroboration_findings, default 20) + raise the fuzzy summary char cap 1500→4000. A corroborating item ranked ≤20 is now caught. Residual (honest): a search that doesn't return the item at all still yields unsupported→needs-review (correct fail-closed; never a false confirmed). No migration.

last_verified: 2026-07-18