ADR-0096: Durable, self-healing monitoring-schedule provisioning

Date: 2026-07-06 Status: Accepted Deciders: Adrian (Soft4U), Claude Opus 4.8

Decision context:

• Latency: none on the request path. The approval hook calls the idempotent ensure_monitoring_schedule() guard-and-swallowed (a Temporal create_schedule, ~10-30 ms, already-exists → no-op) so it never blocks the decision. The reconciler runs out-of-band on a 30-min cron.
• Dependency surface: no new packages — reuses the existing Temporal SDK Schedule client and the already-present reconcile_monitoring_schedules() / ensure_monitoring_schedule() logic. One new system-level Temporal Schedule object.
• Debuggability: strictly improves it — a silent-zero-monitoring state becomes an explicit GET /monitoring/health field and an audit_events row on every heal. Failures of the reconciler workflow are visible in Temporal history, not a swallowed logger.warning.
• Reversibility: single config flip (monitoring_reconcile_interval_minutes) to retune; deleting the monitoring-reconciler Schedule reverts to the prior one-shot behavior. ~4-file change to undo.
• Blast radius: additive. No change to detection checks, alert routing, disposition lifecycle, or next_review_due. Adds one workflow, one activity, one startup call swap, one approval hook, one read-only endpoint.
• Alternative considered: app-side asyncio periodic loop — rejected because it is tied to the app process lifecycle rather than Temporal-durable, and is inconsistent with the per-tenant monitoring loop which is already a Temporal Schedule.

Context

The AMLA perpetual-KYC monitoring loop (ADRs 0083-0088) is fully built: ContinuousMonitoringWorkflow is a per-tenant Temporal Schedule that runs detection checks, writes MonitoringAlert rows (six writers), reads entity_baselines.next_review_due as a live predicate in sweep_due_reviews, and drives the alert disposition lifecycle. The loop is correct when it fires.

The provisioning of the per-tenant Schedule is not durable. A schedule is created at tenant creation and by reconcile_monitoring_schedules(), but the reconciler is invoked only once, at app startup, as a fire-and-forget background task with guard-and-swallow (app/main.py: except Exception → logger.warning), with no retry and no periodic re-run, and there is no hook on case-approval. Consequently, if Temporal was unavailable at boot, a schedule was deleted, or a tenant predates this code, that tenant's monitoring silently stays un-scheduled until the next app restart. For an AMLR Art. 21 obligation this is the dangerous failure mode: the system believes it is monitoring and is not, with no signal. The reconcile logic itself is sound and fail-closed (a config-disabled tenant still gets a paused schedule, never a silent absence) — the defect is purely when it runs and the absence of an observable signal when a gap exists.

Decision

Make monitoring-schedule provisioning durable, self-healing, and observable, reusing the existing reconcile/ensure logic unchanged:

1. A single, tenant-agnostic system Temporal Schedule monitoring-reconciler fires a new ReconcileMonitoringSchedulesWorkflow every settings.monitoring_reconcile_interval_minutes (default 30). The workflow's activity calls reconcile_monitoring_schedules(). ensure_reconciler_schedule() provisions it idempotently at startup, replacing the one-shot fire-and-forget task. A single system schedule means that if provisioning itself fails, it is one loud, health-visible failure rather than N silent per-tenant gaps.
2. An approval hook: on APPROVED / APPROVED_WITH_RESTRICTIONS, call the idempotent ensure_monitoring_schedule(tenant_id) (guard-and-swallowed) so the first approved case guarantees its tenant's schedule immediately.
3. A loud health surface: GET /monitoring/health reports the reconciler-schedule presence and the set of tenants that have an approved case but no schedule; and every reconcile run that had to create a missing schedule (ensured > 0) writes an audit_events monitoring_schedule_healed row.

Consequences

Positive

• Eliminates the silent-zero-monitoring failure mode: a missing schedule self-heals within one reconcile interval without an app restart, and is visible via /monitoring/health in the meantime.
• Healing is audited (AMLR Art. 21 / EU AI Act Art. 12), not swallowed — a compliance record that a monitoring gap existed and was closed.
• Consistent with the existing architecture (the monitoring loop is already a Temporal Schedule).

Negative

• Adds a second always-on Temporal Schedule to operate and monitor. If the reconciler schedule is deleted and the app is not restarted, it does not self-heal (turtles-all-the-way-down) — mitigated by making it a single, health-checked object rather than accepting N silent per-tenant gaps.
• A 30-min interval means a deleted per-tenant schedule can be un-scheduled for up to that window; the approval hook and startup provisioning cover the common cases so the window only applies to out-of-band deletions.

Neutral

• Reconcile frequency is a new tunable (monitoring_reconcile_interval_minutes); the per-tenant monitoring cadence (weekly) is unchanged.

Alternatives Considered

Alternative 1: App-side asyncio periodic loop

• A background task in the FastAPI app re-runs reconcile_monitoring_schedules() every N minutes.
• Why rejected: tied to the app process rather than Temporal-durable, no built-in retry/visibility, and inconsistent with the per-tenant monitoring loop which is already a Temporal Schedule.

Alternative 2: Fold reconciliation into the per-tenant ContinuousMonitoringWorkflow

• Have each tenant's monitoring cron re-ensure its own schedule.
• Why rejected: circular — a tenant whose schedule is missing has no cron to run the self-heal, so the exact failure mode we are closing (a missing schedule) is the one case it cannot fix.

Alternative 3: Do nothing (accept startup-only reconciliation)

• Why rejected: leaves the silent-zero-monitoring failure mode open for an AMLR Art. 21 obligation — the system can be non-compliant (not monitoring) with no operator signal.