Skip to main content

ADR-0197: The frontend OS closure floats too, and an unchanging upgrade layer never patches anything

Date: 2026-08-26 Status: Accepted Deciders: Adrian (project owner), Claude Opus 5, Codex (PR #1213 review) Amends: ADR-0191 (which stays Accepted; only its Status line points here, per s4u-adr §10)

Decision context:

  • Latency: no request-path effect — this is build-time only. Build cost: one apk upgrade (~2 s on a warm index) plus, whenever OS_PATCH_EPOCH changes, a rebuild of every layer after it in the runner stage. On the frontend that is the small tail (user creation, npm removal, artifact copies), not the Next build, which lives in an earlier stage. Not measured on the backend, where the upgrade already ran unconditionally and only its cache key is new.
  • Dependency surface: none added. No package, no action, no base-image change — apk upgrade pulls from the distribution index the pinned base already trusts. What it DOES add is a dependency on Alpine's repository being reachable at build time, which the backend's apt-get update already had.
  • Debuggability: the epoch is echoed into the layer, so docker history answers "when were this image's OS packages last refreshed?" from the artifact rather than from memory. Failure mode is a build-time apk error, loud and local. The BAD failure mode — an operator omitting the variable — is deliberately silent in the image and caught instead by the documented-builds detector.
  • Reversibility: minutes, single-commit. Delete the RUN/ARG pair and the image returns to carrying exactly its pinned base. No data, no schema, no runtime state. The declaration would then need re-deriving, which its own test enforces.
  • Blast radius: every backend, worker and frontend container this repository builds — additive to the runner stage of both Dockerfiles, plus three build.args entries and three documented commands. No application code path changes; a container that never rebuilds is unaffected.
  • Alternative considered: bump the base image digest. Rejected on measurement, not preference — node:22-alpine resolves to the digest already pinned and carries the vulnerable version, so there is nothing to bump to. It becomes correct again once upstream rebuilds, and does not conflict with this.

Context​

ADR-0191 accepted a floating OS closure for backend/Dockerfile and recorded the system ecosystem as measured_at_date. Its Consequences state, as an accepted property of the design, that:

frontend/Dockerfile installs no OS packages and therefore carries exactly what its digest-pinned base carries.

That was true when written. It stopped being true when the frontend image needed an OS patch it could not otherwise reach.

CVE-2026-14456 (OpenSSL, denial of service via unbounded memory, HIGH) turned the Trivy frontend image gate red on master on 2026-08-25, blocking every open pull request. libcrypto3 and libssl3 ship 3.5.7-r0; the fix is 3.5.8-r0. The image pins node:22-alpine by digest in all three stages, and — measured on linux/amd64, the platform CI builds — the pinned digest and the current node:22-alpine tag are the SAME image, both carrying 3.5.7-r0, while Alpine v3.24/main offers 3.5.8-r0. There was no newer tag to bump to. The digest pin fixes the base LAYER; it does not fix the packages available for it, and here that is precisely what kept the vulnerable bytes in the artifact.

A .trivyignore was never a candidate: under ADR-0157 and ADR-0164 the gate goes green because the ARTIFACT changed, never because the scanner stopped looking.

Review then found a second, larger problem — one that predates this change and is the more important half of this record.

Decision​

1. The frontend OS closure floats, on the same terms as the backend's. frontend/Dockerfile runs apk upgrade --no-cache in its runner stage. This amends ADR-0191's Consequences: BOTH images now resolve their OS closure from the distribution's current index at build time, so two builds of the same commit can differ. That is the measured_at_date state ADR-0191 already declares, now true of two images rather than one. NOTICE.inventory.json and NOTICE say so — the declaration is derived from the build inputs, and ADR-0191's own detector (test_the_declaration_matches_what_the_build_inputs_actually_say) FAILED on this change until the record followed, which is what that detector exists for.

2. An unchanging upgrade layer is not a patching mechanism, and both images had one. --no-cache is APK's package-INDEX cache; apt-get update likewise refreshes an index. Neither says anything about Docker's LAYER cache. In both images the OS-upgrade RUN sits before any changing application input, so a rebuild with a warm cache reuses the first successful upgrade indefinitely. The documented production path (deploy/eval/RUNBOOK.md step 4b, $CO build backend worker frontend) is exactly such a rebuild, while CI's runner always starts cold.

The consequence is the claim-vs-check class in its most expensive form: CI would keep reporting a freshly patched image, correctly, about an artifact the deployment host was not building. ADR-0191's stated rationale — "locking the OS closure would stop security patches reaching the image" — was already only half delivered, because the patches were not reaching a cached rebuild either.

Both Dockerfiles now declare ARG OS_PATCH_EPOCH=unset and interpolate it into the upgrade RUN, so the value is part of the command string and therefore part of the layer's cache key. The runbook passes OS_PATCH_EPOCH=$(date -u +%F), so a production rebuild on a new day re-runs the upgrade. Measured, not assumed: with a never-built value the step executes; with the same value it reports CACHED.

The default is unset deliberately. A build that never sets it records os-patch-epoch=unset in its own history — an honest statement that this image's OS packages were never deliberately refreshed, rather than a date implying they were.

Consequences​

Positive​

  • The blocking CVE is fixed by changing the artifact, not by suppressing the scan.
  • The patch mechanism now works on the documented deployment path, not only on a cold CI runner. That gap existed before this ADR and applied to the backend.
  • os-patch-epoch is recorded in image history, so "when was this image's OS last refreshed?" is answerable from the artifact rather than from memory.

Negative​

  • The frontend image's OS closure is no longer reproducible from its digest pin. Two builds of the same commit can differ. Accepted for the reason ADR-0191 gives: a snapshot trades a visible gate failure for an invisible security regression.
  • The mechanism is a handle, not an enforcement. Nothing fails when a build omits OS_PATCH_EPOCH; it records unset and reuses the cached layer. A detector could refuse to deploy an image whose epoch is unset or stale. That is deliberately not built here — it belongs with the deployment gate, not with a CVE fix, and building half of it would be worse than naming the gap.
  • The runbook now carries a build variable an operator can forget. The failure is silent by design (a stale closure, not an error), which is the cost of not making it enforcement.

Neutral​

  • The npm-removal comment in frontend/Dockerfile asserted that npm is "where every Trivy finding against this image came from". This CVE falsified it. Corrected in place rather than deleted, so the record shows what changed.
  • The system COMPONENT versions in NOTICE.inventory.json still read 3.5.7-r0. They are refreshed by the scheduled image re-measure (ADR-0176 moved that off the per-PR path deliberately: it requires building both images), not by this change. The reproducibility declaration, which IS derived from committed files, was updated here.

Alternatives Considered​

Bump the base image digest​

Rejected on measurement, not preference: node:22-alpine resolves to the same digest already pinned, and it carries 3.5.7-r0. There is nothing to bump to. This becomes the right answer again once upstream rebuilds, and does not conflict with the ARG.

.trivyignore the CVE​

Rejected. ADR-0157 and ADR-0164 both establish that the artifact changes or the gate stays red; suppressing a fixable HIGH in a shipped image is the failure those ADRs exist to prevent.

Require --no-cache for production builds​

Rejected as disproportionate. It rebuilds every layer including the Next build and the Python closure, turning a routine release into a long one — and a slow procedure gets skipped, which returns the stale image by another route. The ARG invalidates the one layer that must not be reused.

Leave the cache behaviour undocumented and ship only the CVE fix​

Rejected. It would have shipped a change whose stated purpose is defeated on the documented deployment path, while the PR asserted the patches flow. That is the defect class this repository names most often.