Investigation Confidence Scoring (Pillar 1)

Quantified certainty for every compliance investigation — replacing binary pass/fail with a 4-dimension confidence framework.

Business Value

Compliance officers need to understand not just what the AI found, but how certain it is. Confidence Scoring provides a 0-100 score decomposed into four independently measurable dimensions, enabling evidence-based decision-making.

Architecture

Confidence Dimensions

Dimension	Range	Measures
Evidence Completeness	0-25	Coverage of required document categories
Source Diversity	0-25	Number and variety of independent sources
Consistency	0-25	Agreement between sources on key facts
Historical Calibration	0-25	Accuracy of similar past predictions

Confidence Levels

Level	Score Range	Action
HIGH	85-100	Automated approval eligible
MEDIUM	65-84	Standard review
LOW	40-64	Enhanced review recommended
INSUFFICIENT	0-39	Additional investigation required

Source Diversity — Provenance Recognition (2026-06-30, ADR-0082, #157)

The Source Diversity dimension scores the number and variety of independent sources behind a finding. The OB Holding adversarial validation showed the breakdown mis-attributing provenance, so recognition was hardened:

• Combined provenance is split — a "A + B" provenance string is counted as two distinct sources.
• Real sources are recognised via exact + substring aliases — VIES, localized registries (e.g. Estonian Äriregister), EU sanctions, news / AML adverse-media, and the subject's own website (classified self_declared).
• Internal pipeline stages are not independent sources — mcc_classifier, screening-suppression, financial_analyzer, EVOI, the knowledge graph and country-capability are classified internal_analysis worth 0 pts: they are consulted, not corroborating.
• The L1 honesty principle is preserved — a truly unrecognised source still scores 1 pt (never inflated to a mid-tier default).

The breakdown is computed live by GET /api/cases/{workflow_id}/confidence, so the corrected attribution reflects on refresh without re-persisting. The logic lives in confidence_engine.py and is mirrored to the extracted trustrelay-engines package (#158).

Fail-Closed Penalty on a Material-Check Data Gap (2026-07-03, ADR-0089 Component D)

The OB Holding CRITICAL→medium flip that motivated ADR-0089 had a confidence-layer symptom: a run where the sanctions/PEP/adverse-media screen was data-gapped (no key, provider failure, or the criminal article simply wasn't retrieved) still produced a confidence score in the same range as a run where the screen actually ran and came back clean — a data gap and a genuine clear looked identical to the officer. That is the ADR-0067 "not assessed ≠ clear" contract being violated one layer up the stack, at confidence rather than at the finding.

compute_confidence() (confidence_engine.py) now takes a material_check_incomplete: bool parameter. When True:

• A flat MATERIAL_CHECK_GAP_PENALTY of 20 points is subtracted from the total score, after the reasoning-template confidence_cap/evidence_gate are applied, and clamped at 0.0 — so the penalty always bites regardless of how generous an upstream cap was, and can never go negative.
• confidence_cap_reason is set to "material_check_incomplete" on the returned ConfidenceScore, and a human-readable line — "Material check incomplete (sanctions/PEP/adverse-media data gap) — confidence penalised 20 pts (ADR-0089 fail-closed)." — is appended to the source_diversity breakdown's details, since a missed material screen is fundamentally a source-coverage gap.
• level_from_score() then re-derives the ConfidenceLevel from the penalised total. A 20-point hit is enough to push a borderline HIGH/MEDIUM result down at least one band, so a data-gapped run cannot present as HIGH-confidence "clear" — it structurally can no longer clear the 85-point automated-approval threshold from a starting score below 100, and routes toward MEDIUM/LOW/INSUFFICIENT instead, which the Confidence Levels table already routes to standard/enhanced review rather than automated approval.
• The flag is caller-supplied, not inferred inside the engine: _compute_and_store_confidence() in the workflow (activities.py) passes material_check_incomplete=bool(investigation_result.get("material_check_incomplete")). That field is set upstream by adverse_media_agent.py (no API key, provider failure, or an analysis exception during the material screen) and threaded through osint_agent.py's result dict — the same boolean also fail-floors the entity-baseline reconciliation in entity_baseline_service.upsert_baseline (ADR-0089 Component B) and feeds a material_check_incomplete payload into MonitoringAlertService, so the confidence penalty, the baseline floor, and the alert all key off one upstream signal rather than three independently-inferred ones.

This closes the specific gap the design calls out: a data gap can lower confidence (correctly, since less was verified) but it can never be the reason a downgrade is accepted — Component B's ratchet in entity_baseline_service.py independently refuses to let a material_check_incomplete=True run establish a new, lower baseline. Confidence and baseline risk are reconciled from the same upstream flag but enforce the fail-closed guarantee at two separate layers (presentation and persistence) so neither can be bypassed alone.

Workflow Integration

Confidence scoring is invoked through the _compute_and_store_confidence() helper method, which was extracted from the workflow's run method during the codebase hardening sweep (change I6). This helper is shared between the KYC and KYB investigation paths — both call it after their respective investigation activities complete.

# Shared for both KYC and KYB paths
await self._compute_and_store_confidence(
    input, investigation_result, retry_policy
)

The helper:

1. Checks the confidence-score-v1 version gate (skipped for old workflow histories)
2. Calls the compute_confidence_score activity with a 30-second timeout
3. Appends the result to self._state.confidence_scores
4. Logs a confidence_computed audit event
5. Swallows all exceptions (confidence scoring is best-effort — a scoring failure never blocks case progression)

Prior to I6, confidence scoring was duplicated inline in both the KYC and KYB branches. Extracting it to _compute_and_store_confidence() eliminates the duplication and ensures both paths always score using identical logic.

Key Components

• confidence_engine.py — Core scoring engine with dimensional computation. The ConfidenceScore/ConfidenceLevel Pydantic models live in the shared trustrelay_models.confidence package (ADR-0037); level_from_score() applies the 85/65/40 thresholds.
• calibration_service.py — Feedback loop: officer decisions are recorded via record_data_point() and surfaced via get_calibration_stats(), feeding the Historical Calibration dimension.
• quality_scorer.py — LLM-as-judge quality scoring used alongside the deterministic confidence engine.
• ConfidenceScoreCard.tsx — Visual breakdown in case detail view

API Endpoint

The confidence router is mounted under /api/cases (app/api/confidence.py):

Method	Path	Description
GET	/api/cases/{workflow_id}/confidence	Get the latest 4-dimension confidence breakdown for a case

Calibration is not exposed as a standalone REST surface: officer decisions feed CalibrationService.record_data_point() internally from the decision flow, and get_calibration_stats() supplies the Historical Calibration dimension at scoring time.

Configuration

• The confidence score is computed by the compute_confidence_score Temporal activity and is best-effort: a scoring failure never blocks case progression. There is no dedicated confidence_scoring_enabled feature flag — scoring runs as part of the workflow, gated by the confidence-score-v1 workflow version guard.
• Alembic migration: 006_calibration_data