Skip to main content

ADR-0183: System-scoped reconciliation evidence in the object store

Date: 2026-08-13 Status: Accepted Deciders: Adrian (project owner), Claude Opus 5, Codex (review, PR #1127)

Context​

reconcile_monitoring_schedules() (ADR-0096) sweeps every tenant in one run and repairs missing monitoring schedules. Its output is therefore platform-wide by construction, and two of its findings are lists of subjects rather than counts: schedules whose owning subject no longer exists (orphan), and schedules whose subject has gone stale.

Three constraints collide.

  1. The audit trail has no system scope. AuditService.log_event writes a row under a tenant_id, and the rows are RLS-owned, immutable and hash-chained per tenant (ADR-0064, ADR-0109). tenant_id=None does not mean "the platform" — it falls back to the demo tenant. There is no place in the audit model for a record that belongs to no tenant.

  2. Writing the platform-wide lists into each tenant's rows is a disclosure. The first implementation wrote the same payload into one row per live tenant, so an orphan belonging to tenant C landed in tenant A's and tenant B's immutable chains. That is now fixed: _tenant_safe_report replaces the subject lists with a SHA-256 over the sorted set, so each tenant can prove the reconciliation that covered it matched a set an operator holds, while learning no other tenant's identifiers.

  3. A digest cannot answer the question the sweep exists to answer. In the 460-orphan incident this reconciler was built for, 410 deleted schedule owners were recoverable from nowhere. A digest verifies a set someone already has; it cannot name one. The reports themselves carry only the first 50 of each list.

So the complete lists need a durable home that belongs to no tenant, and the audit trail — the natural home — cannot hold one.

Decision​

Write the complete orphan and stale subject lists to the object store under a system-owned key namespace, and record only the KEY in each tenant's audit row.

  • Key shape: system/monitoring-reconciliation/<UTC-stamp>-<8 hex>.json. The random suffix exists because two runs can start in the same second and must not overwrite each other's evidence.
  • Payload: a versioned object (schema: 1) carrying written_at, the sorted complete orphan_subjects and stale_subjects, their counts, and a why string naming what the artifact is for.
  • Each tenant's audit row carries complete_subjects_artifact: <key> — or an explicit null. Never an omitted field, so "no artifact was needed" and "we did not try" cannot look alike.
  • The write is guard-and-swallow. The reconciliation is the control; a failure to file evidence must never abort a sweep that is repairing monitoring gaps.
  • The complete lists are additionally written to the application log at WARNING before truncation, which is durable enough to answer "which schedules did this run delete?" during an incident and carries no tenant-chain problem.

system/ is hereby the reserved prefix for platform-scoped artifacts that belong to no tenant. No API serves objects under it.

Consequences​

Positive​

  • The question the reconciler exists to answer is answerable. The complete set is retained and addressable, not merely verifiable.
  • No tenant's immutable chain gains another tenant's identifiers, while each tenant still receives a digest that binds its row to the same evidence.
  • The artifact is versioned and self-describing, so a reader five years from now can interpret it without this document.

Negative​

  • A cross-tenant namespace now exists outside the RLS model. Every isolation guarantee in ADR-0023 and ADR-0050 is enforced by Postgres RLS. This object is governed by none of it: anyone holding MinIO credentials can read every tenant's subject ids from one file. That is a genuine widening of the blast radius of a leaked object-store credential.

    The mitigation today is that no endpoint serves objects by caller-supplied key at all — every read path in app/api/ resolves a case or document id and derives the key itself, so system/ is unreachable through the API by construction rather than by omission. That is a stronger statement than "no API serves the prefix", and it is the property to preserve: a future generic file-serving endpoint MUST refuse the system/ prefix explicitly, because the isolation here rests on key derivation and nothing else.

  • The key is visible in every tenant's audit row. A tenant admin reading their own chain learns the key of an object containing other tenants' identifiers. The key is not the data, and the object store is not reachable from any tenant-facing surface — but the pointer crosses a boundary the data does not.

  • Retention is undefined for this prefix. ADR-0140's retention clock and ADR-0108's purge operate on cases and subjects; neither knows about system/. These objects accumulate until something is written to age them out. That is a real gap and is not closed here.

    Backup, however, IS covered, and an earlier revision of this ADR said otherwise. scripts/backup/minio_backup.sh mirrors whole BUCKETS (TR_MINIO_BUCKETS, mc mirror without --remove), so a new prefix inside an already-backed-up bucket is included the moment it exists and rides the same versioning and restore-verification as the case documents. Claiming a backup gap that does not exist is the same defect as claiming coverage that does not — it sends an operator to fix the wrong thing (Codex P2 on PR #1127).

  • Guard-and-swallow means the evidence can silently be absent. The null field discloses it, but nothing alerts on it.

Neutral​

  • The object store already holds case documents and evidence bundles, so no new dependency or operational surface is introduced — only a new prefix.
  • The digest in the tenant rows remains the authoritative binding; the artifact is a convenience for an operator, never an input to a compliance decision.

Alternatives Considered​

Alternative 1: System-scoped rows in audit_events​

Give AuditService a real platform scope so the lists live in the immutable, hash-chained store with everything else.

  • Why rejected: not available without changing the chain model. The chain is per tenant precisely so that an RLS-scoped session can read its own head (ADR-0109). A system chain needs its own sequence, its own head, and a read path no tenant session can reach — a schema and RLS change large enough to need its own ADR and its own migration. Deferred deliberately, not dismissed; it is the correct long-term home, and this ADR should be superseded when it exists.

Alternative 2: Write each orphan under its own tenant​

Record the subject in the audit chain of the tenant that owns it.

  • Why rejected: the tenant row is gone. That is what made the schedule an orphan. There is no tenant to write under, and inventing one would fabricate attribution.

Alternative 3: Keep only the digest (do nothing further)​

Accept that the immutable rows carry a SHA-256 and stop there.

  • Why rejected: measured against the incident it must serve. A digest confirms a set an operator already holds and cannot enumerate one. For the 460-orphan incident, 410 owners were recoverable from nowhere — the exact failure this reconciler exists to prevent recurring.

Alternative 4: Application log only​

Emit the complete lists at WARNING and retain nothing else.

  • Why rejected as the sole mechanism, and partially adopted: the log write is in place and is the reason the residual above is bounded. But a log line is not addressable — an audit row cannot point at it, so no tenant's chain can bind to the evidence, and log retention is shorter than AMLR's five years (ADR-0139).

Decision context​

  • Latency: one additional object-store PUT per reconciliation run, on a schedule that fires every monitoring_reconcile_interval_minutes (default 30). Not on any request path. Not measured, because a single PUT on a half-hourly background sweep cannot be material.
  • Dependency surface: none new. MinIOService is already a dependency of the case-document and evidence-bundle paths.
  • Debuggability: the failure mode is a null field plus a logged exception, and the complete lists are in the application log regardless — so an operator is never left with nothing. Nothing alerts on the null, which is the recorded weakness.
  • Reversibility: high. Deleting _write_complete_subjects_artifact and the one field it populates removes the namespace; no schema, no migration, no data to unwind.
  • Blast radius: additive. One function, one field in a report already written, one new object-store prefix. No existing path changes behaviour.
  • Alternative considered: system-scoped audit rows (Alternative 1) — rejected for now because it requires a second chain with its own RLS read path, which is a larger decision than the gap it would close here.