Skip to main content

ADR-0124: Deterministic, hash-audited risk scoring with frozen reproducibility snapshots

Date: 2026-07-24 Status: Accepted Deciders: Adrian (Soft4U BV), Claude (Opus 4.8) Extends: ADR-0020 (weighted-max EBA matrix), ADR-0089 (entity-risk ratchet), ADR-0092 (audited risk config), ADR-0103 (OpenSanctions bulk), ADR-0109 (append-only audit hash chain), ADR-0123 (deterministic golden-record resolution)

Context​

Epic #511 raises the OSINT engine from real & honest to real, honest & auditable. Issue #515 is the risk-scoring pillar: EBA/GL/2021/02 requires scoring to be versioned configuration, not LLM output, reproducible over time, with reference lists frozen at source + date so a score recomputes identically later.

A full five-map read of the scoring surface established the ground truth (not a greenfield):

  • The EBA scorer (eba_risk_matrix.compute_eba_risk) is already a pure, deterministic function and already emits an input_hash/output_hash. So #515 is not "make scoring deterministic" — it is "make a scoring run reproducible over time."
  • But reproducibility fails five ways: (1) three conflicting MATRIX_VERSION strings, none stamped on the result; output_hash covers only {dimensions, overall_score, risk_level}, with no config_hash and no combined run hash. (2) Reference lists are mutable JSON files loaded into a process-lifetime singleton, and their source/source_url/source_date envelope is stripped at load; risk_configurations is unhashed. (3) The matcher knobs upstream of the matrix are unversioned — four matchers, four threshold sets, two colliding HIT_THRESHOLD values (0.90 vs 0.85), and the OpenSanctions bulk dataset is pulled from a latest/ URL that DELETE+re-INSERTs daily/weekly with no version captured — so the same subject flips hit↔clear across a refresh, invisibly. (4) The LLM risk_score (pre-floor) still drives the AI recommendation in decision_service.py:296, in parallel with the authoritative deterministic EBA risk_assessment. (5) Normalization is deterministic but unversioned — a text-cleaning drift shifts both the survivorship winner and the score hash with nothing to flag it.

The critical, non-obvious insight (product owner): "same matrix version + reference snapshot → identical hash" is a false guarantee unless the matcher config, dataset version, and normalization version are snapshotted too. The structured record fixes provenance, but the match decision still has a knob, and knobs need snapshots.

Decision​

A scoring run reproduces iff every input that can change the output is either hashed or referenced by an immutable snapshot ID. We adopt that boundary and enforce it:

  1. The run hash covers the whole reproducibility boundary, not just the numeric input: run_hash = H(canonical({schema, engine_version, matrix_version, reference_snapshot_ids, matcher_config_version, dataset_snapshot_id, normalizer_versions, inputs})).

  2. One canonical, upgrade-surviving serializer (canonical_json): sorted keys, compact separators, ensure_ascii=False, Decimal → fixed-point strings for every weight/score/threshold (floats are not a stable cross-version contract), and an encoder that raises on non-canonical types (never str()-coerces). Timestamps are excluded from the hashed payload (the golden_record_hash precedent). A schema version guards the canonicalization format itself; an engine_version is hand-bumped on scoring-logic change, never derived from pip freeze, so a dependency patch cannot churn the hash. This replaces the four divergent hash one-liners in the codebase.

  3. Frozen reference snapshots in a new append-only RLS reference_snapshots table (list_key, data, source, source_url, source_date, content_hash), fixing the provenance-stripping at load and at config-embed. Each score records the snapshot IDs it read; a live edit cannot change a past score's inputs.

  4. Matcher-config + dataset snapshots — the reproducibility hole the knobs and the latest/ dataset reintroduce. All matcher thresholds (name-similarity, HIT_THRESHOLD, FP-suppression, pg_trgm cutoffs) plus the rapidfuzz/pg_trgm library versions are frozen as a matcher_config snapshot (matcher_config_version); the OpenSanctions bulk CSV is content-hashed at ingest (dataset_snapshot_id). #515 snapshots these; it does not consolidate the four matchers or resolve the 0.90/0.85 collision (corrective work for #513/#518).

  5. Two-output, versioned normalization: NormalizedValue{canonical, match_key, raw, normalizer_version}. Survivorship compares on match_key (so casing/diacritic variants are not counted as conflicts) but survives canonical (stored/displayed). normalizer_versions enter the run hash.

  6. The deterministic EBA score becomes authoritative everywhere; the LLM risk_score is demoted to a labelled raw_synthesis_risk_score (generalizing the pattern already coded in compliance_case._build_investigation_completed_details). decision_service.py:296 — the one live decision path still keyed to the LLM float — is repointed at the floored EBA composite. All three monotonic ratchets (ADR-0089) and the deterministic escalation floors are preserved; demotion can only raise or hold, never lower.

  7. Source-class protected-field authority (refining ADR-0123/#516): FieldAuthority keyed on a SourceClass enum (classify a source once in the connector contract, not per field), with on_no_authorized_claim="not_assessed" as the explicit fail-closed hinge, and integrity events routed to alerting.

  8. The discipline is enforced by CI, not aspiration: one golden fixture per lever (normalizer / matrix / matcher / snapshots / engine), an end-to-end backstop (green levers + red e2e = an un-versioned input = a determinism bug), a coverage meta-guard (a new lever without a fixture fails to merge), and a fail-closed startup assertion (refuse to boot against a phantom version). Dark-launched behind hash_audited_scoring_enabled.

Decision context:

  • Latency: pure in-process hashing over ~30 fields; sub-millisecond. Snapshot lookups are indexed reads. Not on the scoring hot path in a way that changes p95.
  • Dependency surface: no new packages; reuses hashlib/json/Decimal; one new Alembic table (reference_snapshots) + scoring_runs.
  • Debuggability: a run's hash + snapshot IDs + scoring_runs row make a score fully re-derivable; a CI lever failure names the blast radius and the version to bump.
  • Reversibility: single flag (hash_audited_scoring_enabled=False) restores current behavior. Hours to undo.
  • Blast radius: additive (snapshot capture + run record) + one decision-path repoint (decision_service) + a comparator hardening; behind a flag.
  • Alternative considered: hash only the EBA matrix + reference snapshot (rejected — reintroduces the matcher/dataset/normalization reproducibility hole the owner flagged).

Consequences​

Positive​

  • A score is re-derivable years later from its run hash + immutable snapshots — examiner-defensible per EBA/GL/2021/02 + EU AI Act Art. 12/15.
  • Any determinism-affecting change (weights, list, matcher knob, normalizer, engine) is loud in CI and forces a deliberate version bump with a reviewable diff compliance can sign off.
  • The LLM stops influencing the risk number in the one remaining decision path; the deterministic floor is authoritative.
  • Reference-list provenance (source + date) is captured, not stripped.

Negative​

  • Five version levers whose failure mode is "someone changes behavior without bumping the right one" — mitigated by per-lever CI fixtures + coverage guard + startup assertion, but it is real recurring maintenance friction (the friction is the "intended or bug?" checkpoint).
  • Per-tenant/global snapshot storage grows append-only; a retention/GC policy is deferred.
  • The matcher score-band knobs are snapshotted but dead until #513 makes sanctions_hits structured (OQ2, verified) — the snapshot is correct but partly inert today.

Neutral​

  • The EBA scorer's math is unchanged; only its inputs are now frozen and its output versioned/hashed.
  • Matcher consolidation (the 0.90/0.85 collision, four matchers) is explicitly out of scope → #513/#518.

Alternatives Considered​

Alternative 1: Hash only the matrix version + reference snapshot​

  • Version the EBA weights + freeze the reference lists; leave the matcher and normalization unversioned.
  • Why rejected: the EBA input sanctions_match_type/pep_level is decided by fuzzy matching with tunable thresholds and a daily-refreshed dataset; a threshold or dataset change flips the same subject hit↔clear → the score changes with an identical matrix version + reference snapshot. The guarantee would be false. (Product-owner insight.)

Alternative 2: sha256(pickle.dumps(...)) of the in-memory objects​

  • Hash the Python objects directly.
  • Why rejected: pickle output depends on class definitions + library versions; it churns on every dependency bump and reorder, producing meaningless hash noise that trains people to ignore the hashes. Canonical serialization of semantic inputs is required.

Alternative 3: Keep the LLM risk_score as the primary number with an audit log​

  • Log the LLM's number for traceability without demoting it.
  • Why rejected: an LLM number is non-reproducible and pre-floor; decision_service computing a recommendation from it can under-state a floored-CRITICAL case (OB-Holding: synthesis ~0.83 vs floored 90). The deterministic floor must be authoritative.