Skip to main content

ADR-0143: Declarative versioned OSINT connector data contracts + drift detection

Date: 2026-07-25 Status: Accepted Deciders: Adrian (Soft4U BV), Claude Opus 4.8 (implementation agent)

Decision context:

  • Latency: negligible. Validation is a per-source dict walk over a handful of declared fields against an already-cached reference dataset (one JSON file loaded once at startup, the ADR-0141 pattern). No network, no LLM. Runs alongside the existing parse (observational); it does not add a round-trip.
  • Dependency surface: zero new packages — json/dataclasses/logging/pathlib are stdlib. One new JSON reference dataset + one new pure service module. No model/DB change, no migration.
  • Debuggability: a drift is a LOUD structured log (connector_contract_drift source=… missing=… type_mismatch=…) AND a structured ContractValidationResult carrying each DriftIssue (field / kind / expected_type / actual_type / detail), convertible to a Finding. "Which source, which field, why flagged?" is answerable from the record alone. Every contract carries a version, so "which contract validated this?" is answerable too.
  • Reversibility: additive. The module is new; the JSON is a new file; the survivorship hook is dark-launched behind connector_contract_trust_enabled=False (flag-off ⇒ trust_for is byte-identical). Reverting is deleting two files + one test + the config flag + one edit block in survivorship.trust_for — no data change.
  • Blast radius: additive-with-defaults. The connector parses are untouched (the validator wraps/observes, it never replaces a parse). The only pre-existing test touched is test_reference_data_service.py, whose count/name assertions hardcoded the dataset list (13 → 14). Survivorship winner-selection is unchanged (pinned by a regression test).
  • Alternative considered: encode each source's expected shape as Python constants / Pydantic models next to the parser. Rejected — the point of #518 is that adding or upgrading a source is a reviewable data change, and that a per-field trust manifest lives with the schema as a versioned artifact (EU AI Act Art. 12 reproducibility). The config/reference_data/ envelope already provides source/source_url/source_date provenance and a singleton loader.

Context​

Issue #518 (OSINT auditability epic #511 — the one genuinely-remaining issue). The OSINT connectors (NorthData, the national-registry decoders, GLEIF, VIES) each parse a live provider response into a normalized output dict — a response→field mapping written by hand in each service (gleif_service._parse_lei_record, northdata_scrape_service.to_facts, vies_service, the registry_agent.RegistryAgentOutput shape). When a provider silently changes its API — a field vanishes or is renamed — the parse does not crash; it emits a silent null. Downstream, the survivorship layer (#512/ADR-0123) stamps per-source trust and selects golden-record winners over exactly these fields, so a silent gap in a high-trust source degrades a determination with no signal that anything changed.

Two properties matter:

  1. The source's schema + capabilities + per-field trust must be a versioned, reviewable artifact — not a convention scattered across parsers. Adding/upgrading a source should be a config edit, and a determination should be able to say how much trust each source's each field carries.
  2. Fail-closed / never-suppress (ADR-0067) — a declared field going missing must surface as a loud signal, never a silent null; a response with no contract must be flagged, never silently trusted.

survivorship.py already had the seam: a SOURCE_FIELD_TRUST: dict[(source,field), float] override map (empty by default) consulted by trust_for. Nothing populated it, and the per-field trust weights lived nowhere.

Decision​

Declare each structured OSINT source's schema + capabilities + per-field trust as a versioned config, validate live/normalized responses against it, and alarm on drift — additively, never replacing the parse. Expose the per-field trust so the survivorship layer can use it.

1 — The versioned config (config/reference_data/connector_contracts.json)​

A standard reference-data envelope whose data carries a set version (connector-contracts-v1) and a contracts map. Each source declares its own version, capabilities (the signals it can provide), fields ({name, type, required, allow_empty?, description}), and a per_field_trust manifest (0-1 per field the source asserts). Shipped for the four structured sources whose drift matters most: gleif, vies, northdata, and registry (the shared RegistryAgentOutput shape every run_<cc>_agent honours, so adding a country decoder needs no contract edit). Adding a wholly-new structured source is a new entry in this file — a reviewable data change, no code.

2 — The validator (app/services/connector_contracts.py, pure)​

validate_response(source, response) checks a source's normalized output for a successful fetch against its contract:

  • a declared-required field that is absent OR null → declared_field_missing drift. This is the core case: a vanished/renamed upstream field surfaces as a silent gap, and that is caught.
  • a declared-required field that is present but empty (empty string / empty collection / an object whose values are all blank — e.g. GLEIF drops entity.legalAddress but the parse still emits a five-key address of empty strings; booleans and numbers are never "empty", so valid=False / 0 are real values) → declared_field_missing drift unless the field declares allow_empty. allow_empty decouples two concerns a review (#680) found were being conflated: required (key present + non-null + correct type — catches a renamed/vanished field) vs non-empty (catches a value that collapsed to empty). A field that is legitimately empty in normal operation — a registry roster with zero directors, ubos=[], findings=[] — sets allow_empty:true, so a clean company no longer forces a false HIGH data_source_integrity finding (which feeds the fail-closed verdict), while an absent key still drifts — drift coverage is preserved. registered_address on the registry contract is declared object (it is a dict {street, …} on RegistryAgentOutput), not string, so a real addressed response is not a false type_mismatch.
  • a present field whose type does not match the declaration → type_mismatch drift.
  • a field whose declared type is outside the known set ({string, boolean, number, array, object}) → unknown_declared_type drift (fail-closed — a misspelled type on a config-only source surfaces loudly; it is never silently treated as type-correct, which would disable that field's whole drift check).
  • an unknown source (no contract) → unvalidatable=True (fail-closed — flagged, not silently trusted).
  • a None/empty response → a legitimate not-found (not drift).

On any drift a LOUD connector_contract_drift structured warning is emitted; the result is a typed ContractValidationResult (list of DriftIssue). drift_to_finding() converts a drifted or unvalidatable result into a Finding-shaped dict (category data_source_integrity, HIGH severity, regulatory basis EU AI Act Art. 15) so it plugs into the existing finding mechanism. The validator never mutates the response and never runs in place of the parse.

3 — The per-field trust hook (additive to survivorship, ADR-0123)​

field_trust(source, field) and per_field_trust_manifest() expose the declared trust. survivorship.trust_for gains a dark-launched additive consult behind connector_contract_trust_enabled (default False): an explicit SOURCE_FIELD_TRUST override still wins; then — only when the flag is on — the declared contract trust is consulted; otherwise the flat PROVIDER_TRUST default is returned, exactly as before. Flag-off ⇒ trust_for is byte-identical, so the always-on graph-ETL resolution path is unchanged. The winner-selection logic is untouched — the flag only supplies a declared trust weight for a (source, field); it never changes how winners are chosen. The consult is lazy-imported and fully guarded so a config/import hiccup can never break resolution.

Consequences​

  • Adding or upgrading a source is a reviewable data change (edit the JSON, bump the version) — the connector list is fully data-driven (pinned by a config-only "add a source" test).
  • A provider API change (a declared field goes missing/renames) produces a drift alert (loud log + structured result + optional finding), not silent data loss.
  • The per-field trust is a versioned artifact and an additive input to survivorship — the survivorship winner-selection is not changed in a risky way (additive, dark-launched, reversible).
  • Deferred (tracked follow-up): wiring validate_response into each connector's live parse boundary and routing drift_to_finding into the investigation findings at run time, and flipping connector_contract_trust_enabled on — both Calibration-Review-gated, consistent with the repo's dark-launch discipline. The mechanism (validator + alert + finding + trust hook) lands here, tested end-to-end at the unit level.