ADR-0164: Remove pip — and Only pip — From the Runtime Backend Image

Date: 2026-08-01 Status: Accepted Deciders: Adrian (Soft4U BV), Claude Opus (implementation + measurement), Codex (review findings P1/P2 on PR #956)

Decision context:

• Latency: no measurable runtime impact. The change is one RUN layer in the runtime stage of backend/Dockerfile; it deletes files and imports nothing at container start. Build time grows by the few seconds the layer's own postcondition assertion costs (import setuptools, torch.utils.cpp_extension).
• Dependency surface: strictly negative — this REMOVES a package (pip, ~11 MB including its vendored tree) from the deployment artifact. Nothing is added.
• Debuggability: this is the real cost, and it is recorded in full under Consequences → Negative. A production container can no longer pip install anything, pip list anything, or pip download a wheel for offline inspection. Every one of those has a workable substitute (below), but the ergonomics are worse and an operator who does not know this ADR exists will be briefly confused by pip: command not found in a Python image.
• Reversibility: total, and cheap. Deleting the RUN block restores pip on the next build; there is no data, no schema and no persisted state involved. The exit condition is external: when pip stops vendoring msgpack 1.1.2 / setuptools 70.3.0, or when Trivy stops preferring pip's own SBOM to the filesystem, the block can be deleted with nothing to migrate.
• Blast radius: backend/Dockerfile runtime stage only — which is to say every production backend and worker container, since both run from this image. No application code, no schema, no migration, no runtime flag. It cannot affect a case, a decision, a risk score or an audit record. What it CAN affect is an operator's ability to intervene inside a running container.
• Alternative considered: a .trivyignore entry suppressing the two CVEs. Rejected — see Alternatives. Also considered and rejected: doing nothing and leaving the blocking gate red.

Context

.github/workflows/security.yml runs Trivy against the built backend image and blocks on fixable CRITICAL/HIGH findings (--ignore-unfixed --exit-code 1). That gate failed on two findings:

CVE	Package	Installed	Fixed in
GHSA-6v7p-g79w-8964	msgpack	1.1.2	1.1.3
CVE-2025-47273	setuptools	70.3.0	78.1.1

Neither version is anything this project installs. Measured in the built image:

$ cat /opt/venv/lib/python3.13/site-packages/pip/_vendor/vendor.txt
msgpack==1.1.2
setuptools==70.3.0

They are pip's own vendored copies. pip additionally ships a CycloneDX SBOM at pip/_vendor/bom.cdx.json declaring them, and Trivy reads that third-party SBOM in preference to the filesystem — which is why the findings carry no PkgPath, why the genuinely installed versions (msgpack 1.2.1, setuptools 83.0.0) do not clear them, and why Trivy itself prints Third-party SBOM may lead to inaccurate vulnerability detection alongside the result.

No dependency bump can resolve this. pip 26.2 — the newest at time of writing — still vendors msgpack 1.1.2 and setuptools 70.3.0. The vulnerable bytes are inside pip and they are inside every pip.

So the choice was between suppressing a finding and removing the code it points at.

Decision

Delete pip and its vendored tree from the runtime stage of the backend image. Delete nothing else.

RUN python -m pip uninstall -y pip 2>/dev/null || true \
    && rm -rf .../site-packages/pip .../site-packages/pip-*.dist-info \
              .../ensurepip .../bin/pip* \
    && ! python -c "import pip" 2>/dev/null \
    && python -c "import setuptools, torch.utils.cpp_extension; print(...)"

Three properties are load-bearing:

1. The gate goes green because the vulnerable code is gone, not because the scanner stopped looking. A .trivyignore would have produced an identical green check with the vulnerable bytes still shipped. That distinction is the entire content of ADR-0157, one PR earlier, and applying it inconsistently here would make the principle decorative.

2. The scope is pip only. An earlier revision of this same line also uninstalled setuptools and wheel. That was over-removal: the vulnerable bytes are entirely inside pip's vendored tree, so nothing about the Trivy result required touching either — and it broke a declared dependency for no gain. torch 2.13.0 (pulled in by docling) carries Requires-Dist: setuptools>=77.0.3, and with setuptools gone:

   $ docker run --rm --entrypoint python <img> -c "import torch.utils.cpp_extension"
   ModuleNotFoundError: No module named 'setuptools'

   The rule this ADR sets is the narrow one: delete what carries the CVE, nothing else. Removing an unrelated package because it happened to be in the same pip uninstall argument list is not a security decision, it is a coincidence.

3. The layer asserts its own postcondition. ! python -c "import pip" fails the build if the removal silently stops working; import setuptools, torch.utils.cpp_extension fails the build if the over-removal silently comes back. A Dockerfile comment claiming pip is gone is a claim; these two lines are a check. (The repository has a name for the gap between those: see feedback_claim_vs_check.)

On the reported torch.compile consequence

The review that caught the over-removal stated that it makes torch.compile fail with ModuleNotFoundError: No module named 'setuptools'. In this image that is not what happens, and the record should say so rather than inherit a convenient justification. Measured, with setuptools restored into the image:

$ docker run --rm --entrypoint python <img+setuptools> -c "import torch; torch.compile(lambda x: x+1)(torch.ones(3))"
InductorError: InvalidCxxCompiler: No working C++ compiler found in
torch._inductor.config.cpp.cxx: (None, 'g++')

$ docker run --rm --entrypoint sh <img> -c "which g++ c++ cc gcc; echo exit=$?"
exit=1

torch.compile has never worked in this image and cannot: the runtime stage deliberately installs no compilers (# no -dev packages, no compilers), so the Inductor backend has no g++ to invoke. Restoring setuptools does not change that.

setuptools is retained because torch declares it and an import genuinely breaks without it — a broken dependency set is reason enough — not because it repairs torch.compile. The remedy the review asked for is right; one of its stated reasons is not, and an ADR that launders an unverified mechanism into the permanent record is the same defect class as a scanner suppression.

Consequences

Positive

• The blocking Trivy backend gate passes with 0 fixable CRITICAL/HIGH, with no suppression file anywhere in the repository.
• A running container can no longer install arbitrary code. pip install from inside a compromised container is a standard post-exploitation step; it is now unavailable. This is a genuine hardening win and it would justify the change on its own.
• The image no longer ships an SBOM that misdescribes its own contents. pip's bom.cdx.json asserted msgpack 1.1.2 / setuptools 70.3.0 were present when the real installed versions were 1.2.1 / 83.0.0 — a scanner-visible falsehood about the artifact, now gone.
• Marginally smaller image.

Negative — recovery and debugging inside a running container

This is the cost, stated plainly rather than minimised.

Lost	Substitute
pip install <pkg> — hotfix a library in a live container	Not available, by design. Build a new image. This is the intended posture for an immutable deployment artifact, but it removes an escape hatch that did exist.
pip list / pip show — what is installed?	python -c "import importlib.metadata as m; print(sorted((d.name, d.version) for d in m.distributions()))". Works, uses stdlib, no pip.
pip freeze for a support bundle	Same as above, or scan the image with Trivy/Syft from outside, which is more trustworthy anyway.
pip download for offline inspection	Do it outside the container.
python -m ensurepip as a last-resort restore	Also removed. A truly stuck operator can docker cp a pip wheel in, or exec into a sidecar.

The honest summary: routine inspection is unaffected (importlib.metadata covers it), and in-place mutation is deliberately gone. The failure mode to expect is an operator typing pip in a Python container, getting command not found, and briefly assuming the image is broken. That is why this ADR exists and why the Dockerfile comment names it.

Neutral

• setuptools and wheel remain in the image. Neither carries a fixable CRITICAL/HIGH at the versions installed (setuptools 83.0.0 ≫ the 78.1.1 fix line).
• The builder stage is untouched: it still installs and uses pip normally. Only the runtime stage is stripped.
• If a future dependency genuinely needs pip at runtime, this decision must be revisited rather than worked around — the build-time assertion will not catch that, because it checks that pip is absent.

Alternatives Considered

Alternative 1: .trivyignore the two CVEs

The obvious fix, and the one most repositories take. Rejected: the finding would be suppressed while the vulnerable code stayed in the shipped artifact, and the check would be green for a reason unrelated to the artifact's contents. ADR-0157 — merged one PR earlier — records that a gate turned green for a reason other than the thing it measures is indistinguishable from a gate that ran, and that this repository does not do it. Applying that standard to CodeQL and not to Trivy would make it a slogan.

There is also a narrower objection: the suppression would be permanent in practice. Nobody re-audits a .trivyignore line, and both CVEs would sit there long after pip stopped vendoring them.

Alternative 2: Keep pip; pin msgpack/setuptools to fixed versions

Does not work, and this was measured rather than assumed. The findings are read from pip/_vendor/bom.cdx.json, not from the installed distributions — the image already had msgpack 1.2.1 and setuptools 83.0.0 installed when it failed the gate. Upgrading the real packages changes nothing about pip's vendored copies or the SBOM that declares them.

Alternative 3: Uninstall pip, setuptools and wheel together

What the first revision of this PR did. Rejected on measurement: it broke import torch.utils.cpp_extension (torch declares setuptools>=77.0.3) and cleared exactly the same two CVEs that removing pip alone clears. It traded a real dependency for nothing. Recorded here because the over-removal shipped in a commit and was caught in review, not before it.

Alternative 4: Leave the gate red

Rejected for the reason ADR-0157 spends a section on: a permanently red job for a non-actionable reason is how a gate that does mean something gets routed around. It also masks any genuine future finding — the backend scan would be red either way.

Alternative 5: Switch to a distroless / pip-free base image

The structurally cleaner answer, and probably right eventually: gcr.io/distroless/python3 ships no pip, no shell and no package manager at all. Rejected for now as disproportionate — it changes the base image, the healthcheck (which uses python -c, fine) and the debugging story far more than this does, and it would need its own evaluation of the Pango/Cairo native dependencies WeasyPrint requires. Recorded as the direction of travel, not as work committed to here.

Related

• ADR-0157 — the precedent this follows: a gate must go green because the condition it measures changed, not because the measurement was disabled. Also the ADR that split the Trivy scans into one job each, which is why the backend scan failing no longer silently skips the frontend scan.
• PR #956 — the parent change: the Docker Build CI job was building zero images (every buildable compose service is behind profiles: ["stack"]), and nothing smoke-tested that the image could import app.main. This ADR covers only the package-manager removal in that PR.