ADR-0177: Pin the image's dependency closure, and make the intent a separate file

Date: 2026-08-08 Status: Accepted Deciders: Adrian (project owner — chose "pin requirements.txt" over the cadence option on #1104), Claude Opus 5, Codex (review, PR #1105)

Decision context:

• Latency: not measured, and safe to defer — resolution happens at build time, never in a request path. The one runtime figure that moves is image build duration, and it moves DOWN: a fully-pinned closure removes the resolver's backtracking. Nothing on the serving path changes.
• Dependency surface: the count is unchanged (263 pins; the extras are wheel, which the image already installed unpinned). What changes is the authority: 53 of 56 lines were floors, so the shipped set was decided by whatever PyPI served on build day. Upstream yank: a yanked pinned version makes the build FAIL rather than silently install a neighbour — the correct direction, since a build that cannot reproduce should say so. Residual, recorded not hidden: --index-strategy unsafe-best-match keeps a dependency-confusion exposure that pinning narrows but does not remove, and --generate-hashes (which would close it) is deferred.
• Debuggability: a failure is a resolver error at build time naming the unsatisfiable requirement, on the developer's screen, before anything ships — as against the failure this replaces, which was a scheduled job going red days later against an artifact nobody could reconstruct. requirements.in carries the reason for every floor and cap, so the next reader sees intent, not just numbers.
• Reversibility: minutes. Delete requirements.txt, restore the previous floors, rebuild. No migration, no persisted state, no flag. The generated file is reproducible from requirements.in by one script.
• Blast radius: every backend and worker container — but substitutive, not additive: the same packages at pinned versions. Verified before landing: 258 of 262 components matched the built image exactly, the remainder being three PEP 503 name normalisations and wheel (now closed). CI is a second consumer and was NOT covered — it installed the floating pyproject.toml environment, so tests and pip-audit could pass against a newer release than the image shipped; corrected in this PR.
• Alternative considered: pin requirements.txt as it stood, without splitting intent out. Rejected by measurement — it fixes 1 of the 5 observed drifts, because 4 are transitive and a file of 53 floors cannot express a transitive pin. The cadence option (re-collect NOTICE more often) was rejected by the owner on #1104: it makes the record chase the artifact instead of making the artifact reproducible.

Context

Security / NOTICE matches the built images (scheduled) went red on master (4e2af97b) with ten differences. All ten were the same five packages, each a patch or minor version apart from what NOTICE.inventory.json attributes:

package	attributed	present in the image
docling-parse	7.10.0	7.11.0
hiredis	3.4.0	3.4.1
huggingface-hub	1.26.1	1.27.0
platformdirs	4.11.0	4.11.1
pydantic-settings	2.14.2	2.15.0

ADR-0161 predicted this exact mechanism and named the cause in its own text: rendering is pure over NOTICE.inventory.json so --check can block every PR, "while the image re-measure is scheduled (unpinned requirements.txt)". The scheduled re-measure ran, and it found real drift.

The consequence is not cosmetic. NOTICE and THIRD_PARTY_LICENSES are the files an OEM partner's counsel reads, and they did not describe the artifact that ships. ADR-0161's whole premise is that legal artifacts are collected from the artifact, not asserted about it — a NOTICE that is periodically wrong between scheduled runs is the asserted kind wearing the collected kind's label.

The obvious fix does not work. requirements.txt declared 56 requirements, 53 of them floors (>=). Pinning those 56 fixes exactly one of the five drifted packages — pydantic-settings. The other four are transitive:

docling-parse    <- docling-slim
hiredis          <- redis[hiredis]
huggingface-hub  <- docling-slim, pydantic-ai-slim, transformers, tokenizers, safetensors
platformdirs     <- fastmcp-slim, zeep, griffelib, setuptools

The image installs 262 Python packages. Pinning the 56 direct ones leaves 200+ still floating, so the class survives with 4 of its 5 current instances intact.

Decision

Split intent from resolution, and pin the resolution completely.

backend/requirements.in — new, hand-authored. It is today's requirements.txt verbatim, including its comments: the floors, the caps, and the written reasons for them (notably the pydantic-ai-slim<2 rationale that ADR-0171's parity test exists to protect).

backend/requirements.txt — now GENERATED. A fully-resolved closure of 263 exact pins, compiled from requirements.in. The install COMMAND is unchanged — the Dockerfile still runs pip install -r requirements.txt, and the file is still a valid pip requirements file.

The Dockerfile itself is NOT unchanged: it also pins the build tool, wheel==0.47.0 (backend/Dockerfile:47). That pin is why the image closure is 264 distributions against the lock's 263 — wheel is a build-time dependency the resolved runtime closure does not contain, so it is declared where it is installed rather than folded into a lock that would then misdescribe the runtime set. Saying "the Dockerfile is unchanged" left the canonical record contradicting both the implementation and its own 264-versus-263 explanation three sections down (Codex P2 on PR #1105).

Regeneration:

backend/scripts/compile_requirements.sh                     # add/remove a dependency
backend/scripts/compile_requirements.sh --upgrade           # refresh every transitive pin
backend/scripts/compile_requirements.sh --upgrade-package X # refresh exactly one

Use the script, not the bare uv pip compile this section used to document. Run literally, that command overwrote requirements.txt and DELETED the hand-authored header — the only place the three flags below are explained — and it bypassed the CPU-index refusal, which is the defect the script exists to prevent (Codex, PR #1105). A procedure that cannot be followed literally is not a procedure. The script compiles with the flags below, re-prepends the header, and refuses to write a closure carrying no PyTorch CPU index.

Three flags are load-bearing, and each was arrived at by a failure rather than a preference:

--python-platform x86_64-unknown-linux-gnu — linux/amd64 is the platform that ships, so it is the only closure that is a fact about the artifact (ADR-0171). Resolving on the developer's macOS evaluates sys_platform != "darwin" against the wrong interpreter and skips the torch lines entirely. uv resolves for a target platform without emulation, so this costs nothing.

--index-strategy unsafe-best-match — mirrors pip, which is what the image actually runs. uv defaults to first-index-wins as a dependency-confusion defence; pip considers every index and takes the best version. Compiling under uv's stricter default failed outright: certifi exists on the PyTorch CPU index at a version below the requested floor, and uv refused to look further. A lock produced under a different resolution policy than the installer uses would not describe the artifact, which is the defect this ADR exists to remove. The residual exposure is recorded in Consequences below rather than hidden behind the flag's name.

--emit-index-url — +cpu is a PEP 440 local version that PyPI forbids (ADR-0171), so torch resolves only from PyTorch's own index. uv drops index directives by default. Without this flag the generated file pins torch==2.13.0+cpu with no index that carries it, and every image build fails on an unsatisfiable requirement. Caught by grepping the output, not by assuming.

ADR-0171's parity invariant moves rather than dying

test_dependency_declaration_parity.py compared pyproject.toml against requirements.txt because a deliberate pydantic-ai-slim<2 cap had gone missing from the latter and produced an image that could not import app.main. A generated lock breaks that comparison for no defect — ==1.107.2 against >=1.107.0,<2 fails on every shared package.

The comparisons therefore retarget to requirements.in, which is now the file that carries intent. That keeps the original guard intact and unchanged.

Retargeting alone would have moved a guard and dropped half its coverage, so two tests are added for the failure mode generation introduces:

• test_every_line_in_the_lock_is_an_exact_pin — a lock with a floor in it has not removed the drift class, only hidden it. Fail-closed on a short parse: a lock that reads as zero requirements would make the other check vacuous.
• test_the_lock_satisfies_every_declared_constraint — every constraint in requirements.in must hold against the pinned version. This is what catches a stale lock: bump the cap in the intent file, forget to recompile, and the image installs the old closure while the reviewed file says otherwise. That is ADR-0171's defect one level up.

Only the .in file's own declarations are checked, not the transitive closure — a transitive constraint is the resolver's business, and re-deriving it in a test would be re-implementing the resolver.

Consequences

Positive

• The drift class is removed, not managed. All 263 packages are frozen; a transitive patch release cannot change the artifact without a reviewed diff.
• NOTICE and THIRD_PARTY_LICENSES become stable by construction between deliberate bumps, which is what ADR-0161 wanted and could not have while the input floated.
• Every dependency change becomes a reviewable diff with a named version, rather than something that happens at docker build time.
• The lock is a supply-chain artifact in its own right: what shipped on a given commit is answerable from the repository.

Negative

• Bumps become manual. A security patch to a transitive dependency now requires a recompile and a PR. This is the cost the owner accepted when choosing this option over the cadence option on #1104. Dependabot works with pinned files and can carry most of it.
• requirements.txt is now linux-only. The compile resolves platform markers away, so ; sys_platform != "darwin" is gone from the generated file. This is correct for an image input and matches ADR-0171's split (pyproject.toml is the dev/CI path, requirements.txt is the image path), but anyone who was installing requirements.txt on macOS can no longer do so. No CI job or documented dev flow does.
• --index-strategy unsafe-best-match names a real exposure. With two indexes and best-match resolution, a package published to the PyTorch index at a higher version than PyPI would win. This is pre-existing — it is exactly what pip --extra-index-url already does on every build today — and pinning strictly reduces it, because pip now seeks one exact version rather than the maximum available. It does not eliminate it: an attacker who could publish that exact version to either index would still be served. --generate-hashes closes it and is deliberately deferred (see below).

Neutral

• 939 lines of generated file enter the repository. It is diff-noisy on bumps by design; that noise is the review surface.
• wheel and setuptools remain installed by the Dockerfile directly rather than through this file, so they are absent from the lock. That is unchanged behaviour, recorded so the 264-vs-263 count is not read as a gap: NOTICE.inventory.json carries 264 Python distributions: the 263 pins plus wheel, which IS pinned — in backend/Dockerfile, not in the compiled closure, because it is a build-environment tool rather than a runtime dependency. Measured by normalising both name sets: the difference is exactly {wheel} in one direction and empty in the other.

Alternatives Considered

Re-collect NOTICE on a cadence, leave requirements.txt floating

Manages the symptom. NOTICE is correct immediately after each scheduled run and progressively wrong until the next one, and there is no point at which anyone can say which state it is in. Rejected by the owner on #1104 in favour of removing the class.

Pin only the 56 direct requirements

Cheap and intuitive, and measured insufficient: it addresses 1 of the 5 observed drifts, because 4 are transitive. It would have closed the issue while leaving the mechanism intact — the worst outcome, since the red check would go green.

--generate-hashes

Strictly stronger: pins the artifact bytes, not just the version, which closes the index-confusion residual above. Deferred rather than rejected. Hash pinning interacts awkwardly with a second index serving local-version wheels, and it is separable hardening that does not need to ride the change that removes the drift class. Worth its own ADR once this is stable.

pip freeze from a built image

Captures exactly what ships, which is the right target, but produces a file with no record of why any version is there and no way to regenerate it except by building an image. The .in → lock split keeps the reasoning reviewable.

Verification

• 263 pins, compiled for linux/amd64, both index directives emitted, torch==2.13.0+cpu and torchvision==0.28.0+cpu present.
• 258 of 262 Python components match the built image exactly. The remaining four are 3 PEP 503 name normalisations (jaraco.classes → jaraco-classes) and wheel, which the Dockerfile installs directly. The 7 genuine version differences are the 5 drifted packages plus pydantic-ai-slim/pydantic-graph 1.107.1→1.107.2, all resolving upward to current releases.
• The lock installs. pip install --dry-run -r requirements.txt inside --platform linux/amd64 python:3.13-slim exits 0 and reports Would install … torch-2.13.0+cpu torchvision-0.28.0+cpu …. Real pip, real target platform — the only test that answers the question ADR-0171 poses.
• Parity suite 28 passed. The two new detectors were mutation-tested 4/4 caught, 0 survived: a range in the lock, a lock violating a declared cap, a declared package missing from the lock, and the important one — intent bumped with the lock left stale.