Skip to main content

ADR-0156: Model-Tier Eval-Regression Gate

Date: 2026-07-28 Status: Accepted Deciders: [the engineer + Claude Opus 4.8]

Decision context:

  • Latency: zero on the hot path — the gate is not called during a live investigation. evaluate_tier_change runs only when someone is proposing a tier downgrade (a review-time/CI-time action), and its cost is bounded by however many eval cases + real-model calls the injected runner makes; the gate's own logic (invariant checks, dataclass construction) is pure in-process dict traversal.
  • Dependency surface: no new packages. One new pure module (app/services/model_tier_gate.py), one new versioned reference dataset (config/reference_data/model_tier_eval_set.json, the existing ADR-0141 envelope pattern), and one new function (model_tiers.request_tier_downgrade) plus a TIER_RANK ordinal map in the existing model_tiers.py.
  • Debuggability: fully deterministic — no LLM call inside the gate itself; the runner is an injected callable so a test failure reproduces byte-for-byte with a canned runner. A blocked downgrade raises TierGateNotPassedError naming the exact (agent, from_tier, to_tier) transition with no passing record; a failing case's EvalCaseResult.failures/.error names which invariant failed or which exception the runner raised.
  • Reversibility: single function call to undo — nothing in this PR moves an agent's tier (AGENT_TIERS is untouched), so there is nothing to roll back beyond reverting the file. The gate mechanism itself has no flag (it authorizes nothing until request_tier_downgrade is actually called with a downgrade target), so there is no dark-launch flip needed.
  • Blast radius: additive only. model_tiers.get_model_for_agent/get_agent_tier/AGENT_TIERS are byte-unchanged; the only new code path (request_tier_downgrade) is not called from anywhere in this PR — it exists as the sole legal future path for a downgrade, gated from day one.
  • Alternative considered: no gate, rely on code review alone to catch a bad tier downgrade — rejected, see Alternatives.

Context​

app/services/model_tiers.py (ADR-0029) defines AGENT_TIERS, a static hand-edited dict[agent_name, tier] plus a premium/mid/value/budget tier-to-model map. Several entries carry inline comments recording a past upward correction after a regression was found live — person_validation ("upgraded from value — identity validation is compliance-critical"), task_generator ("upgraded from budget — synthesizes all findings into officer actions"), document_extraction/document_validator ("gpt-5.2 — budget tier drops documents with OCR noise"). Those comments are the only record of why an agent sits where it does, and nothing enforces that the lesson holds going forward: AGENT_TIERS is a plain dict, so a future edit can move any agent — including sanctions_resolver or synthesis, both premium — to a cheaper tier with a one-line PR and no evidence the cheaper model still produces correct compliance-relevant output.

Today this risk is dormant rather than active: premium and mid both resolve to the same underlying model (gpt-5.2), and value/budget are unused (ADR-0029's cost differentiation has never actually been switched on). But that is precisely why a guardrail needs to exist before the day someone flips value/budget to a genuinely cheaper model and starts retargeting agents at it — retrofitting a gate after the first live downgrade regression (the same pattern that produced the three "upgraded from X" comments above) repeats a known failure mode instead of preventing it.

This is #855 (#511 audit P4, part of the OSINT auditability epic's model-governance thread) — a scoped follow-up distinct from the model selection work in ADR-0029 and the outbound-privacy work in ADR-0145 (Model Envelope). It does not touch what model any agent uses today.

Decision​

Build a pure, fail-closed gate that a tier downgrade must pass before it can take effect. This ADR/PR moves no agent's tier — it lands the mechanism only, exactly as the design spec (docs/superpowers/specs/2026-07-28-855-model-tier-eval-gate-design.md) scoped it.

1. Declarative eval set. config/reference_data/model_tier_eval_set.json uses the existing ADR-0141 envelope (list_key/data/source*) and is loaded through ReferenceDataService with a direct-file fallback (the same pattern connector_contracts.py uses). Each case is a fixed input dict plus a small list of invariants — never an LLM-graded score. The closed set of deterministic invariant kinds is field_equals, field_in, min_length, keys_present, forbidden_keys_absent; an invariant declaring anything outside this set is treated as a config defect and fails closed rather than being silently skipped. Seeded agents: mcc_classifier (a wrong MCC on a gambling merchant silently defeats the ADR-0147/ADR-0082 gambling-vertical gate downstream), document_extraction (a dropped UBO row is the OCR-noise failure mode the tier comments already call out), and synthesis (a forbidden_keys_absent invariant over the ADR-0123/ADR-0145 PROTECTED_FIELDS authority keys — a downgrade must never start letting the model assert an authority-only field).

2. The gate — app/services/model_tier_gate.py. Pure dataclasses (Invariant, EvalCase, EvalSet, EvalCaseResult, TierGateResult) and one entrypoint, evaluate_tier_change(agent, from_tier, to_tier, *, runner, candidate_model=None) -> TierGateResult. runner(agent, model_id, eval_input) -> dict (sync or async) is the only place a real model call can happen — it is a plain callable injected by the caller, so the gate's own logic is 100% unit-testable with a canned runner and never touches a real API key in CI. Fail-closed rules, all present from the initial PR and hardened by this review pass (points 3-4 below): no eval cases declared for the agent -> no_eval_cases_declared; a runner exception / non-dict return for any case -> that case fails with the error recorded; any single invariant failure on any single case fails the whole gate (no partial credit, no averaging).

3. Zero-invariant cases fail closed (review fix). A declared EvalCase with an empty invariants tuple previously auto-passed — the empty-list loop in _run_case never appended a failure, so passed = not failures evaluated to True with nothing actually checked. The only thing standing between that config defect and a real downgrade being authorized was the external pin test test_seeded_agents_have_cases, which only covers the seeded dataset, not the gate's own logic. _run_case now checks case.invariants before invoking the runner at all: an empty tuple returns EvalCaseResult(passed=False, failures=("case declares zero invariants (config defect — fail-closed, never a vacuous pass)",)), mirroring the existing unknown invariant type fail-closed branch in _check_invariant. A "tie/unknown/empty" now always fails, matching the same fail-closed posture the rest of the module already applies to a runner crash or a malformed invariant.

4. Eval-set version stamped on the result + checked at read time (review fix). TierGateResult gains eval_set_version: str (default load_eval_set().version at construction, so a hand-built result in a test still lines up with "the current eval set" unless the test deliberately stamps a different one). evaluate_tier_change loads the EvalSet once and stamps its actual version onto both the no_eval_cases_declared result and the normal pass/fail result — so a recorded result always names the exact eval set it ran against. gate_passed_on_record(agent, from_tier, to_tier) now requires both that the recorded result passed and that its eval_set_version equals load_eval_set().version read at call time; a version mismatch returns False with no new evaluation run. This closes a staleness gap: without it, strengthening the eval set (adding a case, or tightening an existing invariant, in response to exactly the kind of regression this gate exists to catch) would not retroactively invalidate an already-recorded in-process pass from before the strengthening — a downgrade authorized under the old, weaker eval set would keep sailing through request_tier_downgrade under the new one with no re-run.

5. Recording + enforcement precondition. record_gate_result/get_recorded_gate_result/gate_passed_on_record keep an in-process record of the latest outcome per (agent, from_tier, to_tier) triple (mirrors the existing _tier_models module-level-state pattern already used by model_tiers.py; a durable/DB ledger is explicitly out of scope — no migration). model_tiers.py gains TIER_RANK (ordinal premium > mid > value > budget, an unknown tier ranking below every known tier so an unranked move counts as a downgrade rather than silently passing) and request_tier_downgrade(agent_name, to_tier) — the sole programmatic mutator of AGENT_TIERS. A lateral move or an upgrade is applied unconditionally (it can only ever raise or hold scrutiny steady); a downgrade raises TierGateNotPassedError and leaves AGENT_TIERS untouched unless gate_passed_on_record(...) is True for that exact transition. AGENT_TIERS itself is unchanged by this PR — this function exists so a future downgrade has exactly one legal path, and that path is gated from the moment it is introduced.

Non-goals (unchanged from the design spec): no agent's tier moves in this PR; no real-LLM eval runner wired into CI (the runner is pluggable/mockable by design — a REAL_LLM=1-style opt-in runner is a follow-up, not built here); no DB-persisted gate-result ledger; no automatic CI job invoking evaluate_tier_change on every PR (that wiring is Calibration-Review-gated future work, once a real runner exists).

Consequences​

Positive​

  • A future tier downgrade of a judgement-adjacent agent (sanctions, synthesis, MCC classification, document extraction) has exactly one legal path, and that path cannot be exercised without a passing, currently valid eval-regression result on record — closing the gap the three "upgraded from X" tier comments show was previously only caught after a live regression.
  • The gate is 100% unit-testable with zero real API spend (the runner seam), so CI can exercise the fail-closed logic exhaustively without a model budget.
  • Both hardening fixes in this pass close genuine "silent pass" holes: a config defect (empty invariants) and a staleness hole (eval-set strengthening not retroactively invalidating a stale pass) each independently could have let an unverified downgrade through the gate that exists specifically to prevent that.
  • Extending the eval set is a reviewable data change (edit the JSON, bump version) — no code change needed to add a case, and the version-check makes bumping version load-bearing rather than cosmetic (a bump immediately invalidates every stale recorded pass).

Negative​

  • The in-process record (_RECORDED_RESULTS) is not durable — a process restart between running evaluate_tier_change and calling request_tier_downgrade loses the recorded pass, requiring a re-run. This is a known, explicitly out-of-scope limitation (design spec §3); a DB-backed ledger is future work.
  • The seed eval set is small (three agents, five cases total) and covers only the invariant types listed above. It does not exercise every judgement-adjacent agent in AGENT_TIERS (e.g. sanctions_resolver, citation_semantics, adverse_media have no seeded cases yet), so a downgrade attempt on an un-seeded agent fails closed via no_eval_cases_declared rather than via a meaningful eval — correct behavior, but it means "no cases" and "cases exist and pass" are the only two states available for most agents today.
  • No real-LLM runner exists yet. The gate's fail-closed logic is fully tested, but nobody has actually run a candidate cheaper model through it end-to-end — the gate is unproven against a real regression until that runner is built and exercised (tracked as Calibration-Review-gated follow-up, matching the pattern used for other dark-launched controls in ADR-0146).
  • AGENT_TIERS remains directly editable as a plain Python dict — request_tier_downgrade is the documented path, not a structurally enforced one; a reviewer who edits AGENT_TIERS[...] = "budget" inline bypasses the gate entirely. Code review is the only guard against that until a stronger enforcement point exists (e.g. a lint rule or a frozen-mapping wrapper), matching the caveat already recorded in model_tiers.py's module docstring.

Neutral​

  • No agent's tier changes in this PR — AGENT_TIERS, get_model_for_agent, get_agent_tier are byte-identical to pre-#855 behavior.
  • No migration, no new config flag, no dark-launch toggle — the mechanism has nothing to activate; it becomes load-bearing only the first time someone calls request_tier_downgrade with an actual downgrade target.

Alternatives Considered​

No gate — rely on code review alone​

The status quo before this PR. Rejected: the three "upgraded from X" comments already present in AGENT_TIERS are direct evidence that a plain-dict tier edit reviewed by a human missed a real regression at least three times before being caught live and corrected. Code review catches an obviously wrong tier choice; it does not catch a cheaper model that still runs and still returns a plausible-shaped output but silently gets the gambling-vertical MCC code or a UBO percentage wrong.

LLM-graded eval ("does this output look right?")​

Have a judge model score each candidate-model output for quality instead of checking fixed deterministic invariants. Rejected: non-deterministic by construction — the same candidate output could pass or fail the gate on different CI runs, and a temperature-independent LLM judge is exactly the pattern ADR-0121 (semantic citation verification) had to work hard to calibrate away from false positives/negatives for a much narrower problem. A tier-downgrade gate that itself needs its own regression-testing is the wrong shape for a control whose entire job is fail-closed determinism; the deterministic-invariant design mirrors the existing agent test assertions (test_model_tiers.py's substring checks, mcc_classifier's mock-mode branch logic) rather than inventing a new soft-scoring mechanism.

DB-persisted gate-result ledger (durable, cross-process)​

Persist TierGateResult rows to a table instead of an in-process dict, so a recorded pass survives a process restart and is auditable historically. Rejected for this PR specifically because it requires a migration and a decision about retention/RLS scoping that is better made once a real-LLM runner exists and the gate is actually being exercised against live candidate models — building the durable ledger before there is any real data to persist would be speculative. The in-process record is explicitly named as a placeholder for this in the design spec; tracked as follow-up, not a rejection of the idea.