ADR-0112: Hierarchical (shrinkage) fallback for the Historical Calibration dimension

Date: 2026-07-14 Status: Accepted Deciders: Adrian (Soft4U), Claude Opus 4.8 Related: ADR-0089 (deterministic compliance scoring + entity-risk ratchet — its calibration path is the implicit strict-per-scope decision this ADR supersedes), ADR-0067 (fail-closed / "not assessed" — never report clear when no check ran), ADR-0020 (EBA risk matrix), #239

Context

The Historical Calibration dimension (1 of the 4 confidence dimensions, 0-25 pts) measures how often a compliance officer has historically agreed with the engine's recommendation, and is scored strictly per (officer, template, country) scope with a 10-decided-case warm threshold (compute_historical_calibration). Below 10 in-scope decisions the scope is "cold" and pinned at the neutral 12.5/25.

That per-scope strictness has a structural failure mode. The compliance gates that make the product correct — EDD/UBO/SAR routing, four-eyes, the fail-closed material-check contract — all starve terminal-decision volume: comparatively few cases reach a recorded officer decision, and those that do fragment across (officer × template × country). So nearly every scope stays cold indefinitely, and the dimension yields no signal exactly when it would be most useful (a new officer, a new jurisdiction). Meanwhile real, accumulated cross-scope evidence — "this officer agrees with the engine 71% of the time across everything they've decided" — is invisible to the score.

The naïve fix (pool all history) is a compliance hazard: it would let a favourable organisation prior silently raise a thin officer's confidence above what their own evidence supports — a scrutiny-reducing output with no evidence trail, exactly the class ADR-0067/0089 exist to prevent.

Decision

Add a hierarchical empirical-Bayes shrinkage fallback to the calibration dimension: compute_historical_calibration_shrinkage(scopes: CalibrationScopes). It blends calibration evidence from the broadest scope down to the narrowest, pulling each level toward the running prior with an inverse-sample-size weight, so a cold in-scope scope inherits a disclosed, conservatively-capped borrowed prior and converges to its own specific rate as volume builds.

Scope hierarchy (broadest → narrowest)

L4 tenant → L3 tenant+country → L2 officer → L1 officer+country → L0 officer+template+country

CalibrationService.get_calibration_scopes(officer_id, case_type, country) fetches all five as (n, agreed) pairs. L4/L3 use a new tenant-scoped get_calibration_stats_tenant(country="") (no officer predicate — calibration_data_points carries a tenant_id column + an RLS policy, migrations 016/017, so the tenant session scopes the count automatically; there is no cross-tenant leak). When no country is supplied, L3/L1 collapse onto L4/L2 and are emitted empty so the blend never double-counts the same evidence.

Algorithm (deterministic, temperature=0 — no LLM)

K = 10 (pseudo-count == the warm threshold), NEUTRAL_RATE = 0.5. Starting from the neutral prior, for each non-empty level broadest→narrowest: posterior = (n·p + K·prior) / (n + K); the posterior becomes the running prior. Each contributing level is recorded (label, n, rate) for disclosure. No randomness, no model call — identical inputs → identical score (ADR-0089).

Conservatism cap (never-inflate — the load-bearing compliance rule)

Applied after the blend, keyed on the in-scope L0 sample size n_narrow:

• n_narrow == 0 (borrowed): no in-scope evidence → effective_rate = blended_rate (the borrowed prior). maturity = "borrowed", borrowed = True. If there is nothing anywhere to borrow, it stays the honest cold 12.5.
• 0 < n_narrow < 10 (thin): effective_rate = min(blended_rate, p_narrow, prior_rate_before_L0) — the more-conservative (lower) of the officer's own in-scope rate and the borrowed prior. A favourable org prior can never raise a thin officer above their own evidence. This can dip below neutral 12.5 — that is the fail-toward-scrutiny direction (lower confidence ⇒ more scrutiny) and is intended.
• 10 <= n_narrow (warm/mature): the in-scope scope already dominates; delegate to the existing strict compute_historical_calibration(n_narrow, agreed_narrow) so behaviour is byte-identical to the pre-shrinkage engine.

score = round(min(effective_rate · 25, 25), 2).

Invariant (tested): for a thin scope whose own rate is the more conservative, the shrinkage score ≤ the strict-path score. The fallback can only lift a scope out of the cold dead zone (n_narrow == 0), never inflate a scope that has its own (worse) evidence.

Disclosure (provenance — never present borrowed as in-scope)

HistoricalCalibrationBreakdown gains scope_provenance: str, contributing_scopes: list[dict], and borrowed: bool (all backward-compatible defaults). total_cases/agreement_count stay the in-scope counts; borrowed evidence is surfaced only in contributing_scopes and flagged by borrowed + a human details line ("…disclosed as borrowed, not in-scope evidence"). A borrowed prior is never folded into the in-scope numbers.

Configurable-off

New calibration_shrinkage_enabled: bool = True. When False, compute_confidence uses the strict per-(officer,template,country) path on the L0 counts alone (current behaviour) — for regulators requiring strictly per-jurisdiction calibration. Both runtime call sites (app/api/confidence.py, app/workflows/activities.py::compute_confidence_score) fetch the 5-level scopes and pass the flag from config; legacy 2-int callers are unchanged.

Decision context:

• Latency: get_calibration_scopes issues up to 5 small aggregate COUNT(*) queries (2 fewer when country-less) against an indexed, tenant-scoped table; negligible, no network/LLM calls.
• Dependency surface: none new — a frozen dataclass + arithmetic in the existing engine, and one extra SQL aggregate in the existing service.
• Debuggability: the blend is a pure function; every contributing level and its (n, rate) is disclosed on the breakdown, and a borrowed score is explicitly labelled — the score is fully reconstructable from the breakdown.
• Reversibility: calibration_shrinkage_enabled=False restores the exact strict path; the flag is the kill switch. No schema/migration, so rollback is a config flip.
• Blast radius: only the Historical Calibration dimension (≤25 of 100 confidence points); confidence never gates the deterministic risk decision. The never-inflate cap means the change can only lower or hold a thin/cold scope relative to a naïve pool, never raise it above its own evidence.
• Alternative: see below.

Consequences

Positive

• A previously-cold officer/jurisdiction gets a disclosed, evidence-backed calibration signal instead of a permanent neutral 12.5 — the dimension does its job when it matters most.
• Never-inflate is enforced structurally (the min(...) cap), and every borrowed value is transparent and officer-inspectable — aligned with the ADR-0067/0089 "add scrutiny, never suppress a signal" principle.
• Deterministic and self-documenting: the breakdown carries its own provenance.

Negative

• More moving parts in the calibration path (five queries, a blend, a cap) than a single lookup — mitigated by the pure-function design and full test coverage.
• A thin unfavourable scope can now score below 12.5, which reads as "worse than no information." This is deliberate (fail-toward-scrutiny) but must be understood by officers reading the score; the details line explains it.

Neutral

• total_cases/agreement_count remain in-scope-only; consumers that displayed those unchanged see no difference. The three new fields default to the strict-path values, so any code path that never supplies scopes is byte-identical to before.
• Tracked debt: the drifted trustrelay-engines package copy of compute_historical_calibration is not reconciled here — it is unused at runtime (both call sites import the backend app.services.confidence_engine copy). Reconciling the two copies is deferred (YAGNI); noted so the divergence is not mistaken for a bug.

Alternatives considered

1. Pool all history flat (single global rate per officer). Rejected — it lets a favourable organisation/officer-overall prior raise a thin scope's confidence above its own evidence, a scrutiny-reducing output with no cap and no provenance. This is precisely the defect ADR-0067/0089 guard against. Shrinkage with the conservatism cap keeps the recall benefit while making inflation impossible.
2. Lower/remove the 10-case warm threshold. Rejected — it treats 2-3 in-scope decisions as "mature," making the score volatile and over-confident on almost no evidence, and still ignores the cross-scope evidence that actually exists. It trades one honesty problem for a worse one.
3. Leave it strict (status quo). Rejected — the dimension is effectively dead for the majority of scopes given how the compliance gates starve terminal-decision volume; the accumulated cross-scope evidence stays invisible.

Supersession

Supersedes the implicit strict-per-scope calibration decision embedded in ADR-0089's calibration path: below the warm threshold the dimension no longer always rests at the neutral 12.5 — it may rest on a disclosed, conservatively-capped broader prior. ADR-0089's determinism and never-suppress guarantees are preserved and extended (the borrowed value is disclosed and cannot inflate). ADR-0089 remains Accepted; this ADR narrows its calibration sub-decision.