ADR-0171: Ship docling on the CPU-only torch build

Date: 2026-08-02 Status: Accepted Deciders: Adrian (Soft4U BV), Claude Opus 5

Context

docling is how every uploaded customer document becomes markdown (step 8 of the investigation loop). It runs layout and OCR models on torch, which arrives unconditionally: docling-ibm-models declares torch<3.0.0,>=2.2.2 and torchvision<1. Nothing in this repo asked for torch, and nothing pinned which build of it we got.

On linux/amd64 — the platform the deployed image is built for — the default torch wheel resolves the CUDA runtime alongside it. Measured on a real amd64 interpreter, resolving backend/requirements.txt:

	distributions	torch	NVIDIA/CUDA/triton
before	279	2.13.0	19
after (this ADR)	260	2.13.0+cpu	0

Those 19 are cuda-bindings, cuda-pathfinder, cuda-toolkit, nvidia-{cublas,cuda-cupti,cuda-nvrtc,cuda-runtime,cudnn-cu13,cufft,cufile,curand,cusolver,cusparse,cusparselt-cu13,nccl-cu13,nvjitlink,nvshmem-cu13,nvtx} and triton. All 21 rows (those plus torch and torchvision) are in the committed NOTICE.inventory.json, so they are in the artifact we hand a partner.

Their licences are the problem, not their size. They carry four distinct proprietary strings — LicenseRef-NVIDIA-Proprietary (6 packages), Other/Proprietary License (8), LicenseRef-NVIDIA-SOFTWARE-LICENSE (cuda-bindings), NVIDIA Proprietary Software (nvidia-cusparselt-cu13) — plus cuda-toolkit, whose licence field is null. LicenseRef- is the SPDX marker for "not a recognised open-source licence, bespoke terms attached". In an SBOM for a container an OEM partner redistributes under their own brand, those are the rows their counsel stops on.

And we never execute any of it. There is no GPU in the deployment target, no CUDA device for torch to find, and no code path that asks for one. We were shipping a proprietary GPU runtime, and attributing it in our legal artifacts, to run inference on the CPU.

Docling has no lighter install. Checked rather than assumed: its extras are asr, easyocr, htmlrender, ocrmac, onnxruntime, rapidocr, remote-serving, tesserocr, vlm, xbrl. Every one is additive; none subtracts torch, because the requirement is not conditional in the first place.

Decision

Pin the +cpu build of torch and torchvision, sourced from PyTorch's own CPU wheel index, in both backend/requirements.txt (the image's input) and backend/pyproject.toml (what dev and CI install). The two declarations are byte-identical, including the environment marker.

--extra-index-url https://download.pytorch.org/whl/cpu
torch==2.13.0+cpu; sys_platform != "darwin"
torchvision==0.28.0+cpu; sys_platform != "darwin"

Three details are load-bearing:

The marker. +cpu wheels are published for manylinux x86_64, manylinux aarch64 and Windows — but not for macOS, where Apple never had CUDA and the default wheel already is the CPU build. An unconditional pin would make both dependency files unresolvable on a Mac. Off-marker, torch arrives transitively from docling, which is the correct wheel there.

One declaration per name. A second, darwin-pinned line was drafted and rejected: the parity suite's index holds one Requirement per name, so two torch entries would make every parity comparison silently read only the last one and report an agreement the files did not have. The suite caught this.

Identical in both files. test_shared_packages_declare_the_same_version_constraint and ..._environment_markers refuse any divergence, on the grounds that the image installing 2.13.0+cpu while dev and CI resolve something else means the deployment artifact runs code nobody tested. A first draft put a floor (torch>=2.2.2,<3) in pyproject and the exact pin in requirements; that is exactly the divergence those tests exist to catch, and satisfying them was the right move rather than teaching them to tolerate a local-version suffix.

Because +cpu is a PEP 440 local version — which PyPI forbids — it resolves only from PyTorch's index. PIP_EXTRA_INDEX_URL is therefore set on the three CI steps that run pip install -e, scoped to the step rather than the workflow. Absent it, resolution fails loudly; it can never silently fall back to the CUDA wheel this ADR exists to remove.

Consequences

Positive

• 19 distributions leave the amd64 closure; the diff removes only those 19, adds nothing, and changes no other package's version.
• Four proprietary licence classes and one null-licence package leave NOTICE, THIRD_PARTY_LICENSES, NOTICE.inventory.json and the SBOM.
• The backend image goes 3.44 GB -> 893 MB on linux/amd64 (74% smaller), and a full Trivy scan takes 44 s with 0 fixable CRITICAL/HIGH. Both figures are docker image inspect .Size on images built the same way from the same Dockerfile — the pre-existing local images are arm64, which is not the platform that ships and therefore not a valid comparison. The before-image was confirmed to genuinely carry the stack it is being compared for: 287 distributions, torch 2.13.0+cu130, cuda_compiled 13.0, 19 GPU packages.
• Which torch build ships is now a recorded decision instead of a resolver side-effect.

Negative

• --extra-index-url widens the resolution surface: pip consults the PyTorch index for EVERY name, not just torch, and takes the highest version it finds across both. Measured: that index hosts 96 projects, including certifi, requests, urllib3, setuptools, numpy, pillow, jinja2, filelock, sympy and networkx — around twenty of which are real runtime dependencies here. An earlier draft of this ADR said the index "hosts only torch-family packages, so the practical exposure is small". That was false, and it was the premise the risk acceptance rested on; corrected in review of PR #997. The residual risk is that a package we depend on resolves from a second host without anyone noticing. It is bounded by the index being PyTorch's own infrastructure (same trust class as PyPI, not an attacker-controlled mirror), and pip offers no per-package index scoping — --index-url would REPLACE PyPI entirely, which is worse. Recorded as accepted-and-monitored, not as small.
• Any environment installing the backend without PIP_EXTRA_INDEX_URL set now fails. That is deliberate (loud beats silent), but it is a new setup step for a Linux developer, and it is recorded in both dependency files rather than in someone's memory.
• The pin does not bind on macOS. A Mac gets whatever docling resolves, so local extraction behaviour is not pinned by this change — only the artifact's is.
• Deploying to GPU hardware later requires deliberately reversing this, not just provisioning a GPU. That is the intended trade: it should be a decision.
• torch and torchvision are now version-pinned exactly, so a docling release requiring a newer torch will conflict until the pin is bumped. Previously the resolver absorbed that silently.

Neutral

• No application code changes. docling_service.py is untouched; docling's own device selection already falls back to CPU.
• frontend/Dockerfile and the frontend closure are unaffected.

Verification

The closure must be verified on linux/amd64, and only there. Two other platforms produce a confident false pass, for different reasons, and both were hit during this work:

• arm64. The build must target the platform that ships. A local arm64 build was started and killed on the belief that arm64 "publishes no NVIDIA wheels at all, so it reads clean before the change as well as after" — and that belief was wrong: this repository's own pre-change inventory, collected on linux/arm64, contains all 19 GPU distributions. The measurement would not have been vacuous; it would have been a measurement of the wrong artifact, which is a different and less interesting mistake. Corrected in review of PR #997. The conclusion stands for the honest reason: amd64 is the deployed platform, so it is the only closure whose contents are a fact about what we ship.
• macOS. sys_platform markers are evaluated against the running interpreter, not pip's --platform flag. Resolving on a Mac silently skips the torch lines entirely, so the resolve would not even exercise the pin.

Both the before/after resolve and the document round-trip were therefore run inside --platform linux/amd64 containers.

The round-trip is the check that actually matters. "The image builds" and "the suite is green" would both pass while extraction silently degraded, because docling falls back rather than raising — a degraded path looks like a working one until a partner runs a real PDF. So the same two documents were pushed through the app's own DoclingService (not a raw DocumentConverter — the service configures the pipeline, and it is the service's behaviour that ships) under each torch build, and the markdown was hashed:

	before	after
torch	2.13.0+cu130 (cuda_compiled 13.0)	2.13.0+cpu (cuda_compiled None)
03_company_articles_of_association.pdf	855b1abd4c28fa98… 1604 chars, 61.2 s	855b1abd4c28fa98… 1604 chars, 60.6 s
04_ubo_register_extract.pdf	b197e7757c549765… 1209 chars, 9.2 s	b197e7757c549765… 1209 chars, 6.8 s

Byte-identical output, no measurable slowdown.

Correction, from review (PR #997). A first version of this section claimed the OCR path "genuinely ran" on the strength of RapidOCR logging Using engine_name: torch / Using CPU device and loading its 770 weights. That was wrong. Those lines prove the engine initialised; they do not prove an image page was processed. Both PDFs were then checked and contain zero /Subtype /Image XObjects, so they exercise the layout model only — DoclingService configures force_full_page_ocr=True on InputFormat.IMAGE, and that pipeline was never entered. The probe's own docstring says "a probe that only reads PDFs half-proves the swap", and the first run did exactly that.

An image was therefore added to both sides — a rendered certificate page (backend/tests/fixtures/ocr/, synthetic content, no real entity) which forces the InputFormat.IMAGE branch, since only a pixel page does:

	before (2.13.0+cu130)	after (2.13.0+cpu)
05_scanned_incorporation_certificate.png	9215f916… 350 chars, 59.9 s	9215f916… 350 chars, 57.5 s

Byte-identical on the OCR path too. The fixture is checked to be non-degenerate before the match is trusted: an image that OCRs to nothing would hash the same on both sides and read as a clean pass — the same false-pass shape this section exists to prevent. It yields 350 characters beginning ## CERTIFICATE OFINCORPORATION, and the run-together words are the evidence that this is real character recognition rather than a text layer being read out of the file.

Reproduce with scripts/docling_roundtrip_probe.py. Its default output is structural only — sha256, character count, line count, timings. No document text at all, not even excerpts: head/tail/markdown are all behind DOCLING_PROBE_INCLUDE_MARKDOWN=1, because the probe is meant to be pointed at real uploaded KYC/KYB documents and the JSON gets kept, attached to PRs and pasted into reviews. An earlier revision of this paragraph said head/tail were written by default, which stopped being true when the excerpts moved behind the opt-in — a reproduction instruction promising diagnostic output the tool does not produce (corrected after Codex flagged it on PR #1016).

What the round-trip does NOT establish

It compares torch build A against torch build B in an environment where docling can run. It does not establish that the shipped backend image can run docling — because it cannot, and that predates this ADR.

Running the same probe inside the built image (both the pre- and post-change amd64 builds) fails identically on every document:

ImportError: libGL.so.1: cannot open shared object file: No such file or directory

opencv-python, pulled in by docling/RapidOCR, links libGL.so.1; the runtime stage is python:3.13-slim, which does not ship it, and ldconfig -p finds it in neither image. The CUDA wheel never supplied it either — it is an OS package, not a Python one — so this is orthogonal to the torch build and is tracked separately as #998, deliberately not folded in here (fixing it adds OS packages, which moves the --system closure and forces a NOTICE re-collect).

Worth recording how it stayed hidden: the verification container for this ADR installs libgl1 and libglib2.0-0 itself, and that line went unquestioned until the probe was pointed at the real artifact. A proxy environment that repairs the thing it is standing in for cannot detect that the real one is broken — name the oracle, and prefer the artifact that ships.

What this does not remove

THIRD_PARTY_LICENSES goes from 194 NVIDIA/CUDA/triton mentions to 16, not to zero, and the residual is worth stating precisely because the headline number invites an overclaim.

The 16 are not distributions. Fourteen sit inside torch 2.13.0+cpu's own bundled notice text — NVIDIA copyright lines and one LicenseRef-NvidiaProprietary header covering source vendored into torch itself (CUTLASS and similar). One is an X.Org-era NVIDIA copyright line in libpixman-1-0; one is in pyarrow.

Redistributing the CUDA runtime as 19 separately-licensed distributions is a different legal fact from carrying attribution text inside one BSD-3-licensed package. The former is what an SBOM reviewer flags and what this ADR removes; the latter cannot be removed without removing torch, and is correctly attributed where it is.

Alternatives Considered

Alternative 1: Keep the CUDA stack and write the licensing paragraph

• Explain in the data room why the SBOM carries a proprietary GPU runtime.
• Why rejected: it is an acceptable fallback and was the stated position if docling could not work on CPU-only torch — but docling can, so the paragraph would exist only to explain 19 proprietary rows and one null-licence package for a stack that never executes. Removing them is cheaper than defending them, and a shorter true story beats a longer one.

Alternative 2: A docling extra that avoids the GPU path

• Install docling with a lighter extra and skip torch entirely.
• Why rejected: no such extra exists. All ten extras were read from docling's packaging metadata; every one is additive, and torch comes from docling-ibm-models unconditionally.

Alternative 3: Suppress the findings rather than change the artifact

• A .trivyignore entry, or an SBOM filter, to stop the proprietary rows being reported.
• Why rejected: same precedent as ADR-0164 and ADR-0157 — the gate goes green because the artifact changed, never because the scanner stopped looking. A suppressed row is still redistributed under NVIDIA's terms.

Alternative 4: Pin only in requirements.txt, leave pyproject on a floor

• Let the image pin the build and let dev/CI resolve whatever satisfies the floor.
• Why rejected: the parity suite refuses it, correctly. The image would ship a torch that dev and CI never exercised.

Decision context

• Latency: not measured as a p50/p95 delta, and no regression is expected — there is no GPU in the deployment target, so the CUDA build was already running every docling inference on the CPU. What is measured is the extraction output itself (see Verification), which is the property that matters here.
• Dependency surface: −19 distributions, −4 proprietary licence classes, −1 null-licence package. Adds one extra package index to the resolution path.
• Debuggability: a missing index fails at install time with an unsatisfiable requirement, naming the pin. There is no runtime failure mode: docling selects its device at load and already logged Using CPU device before this change.
• Reversibility: one commit — revert two dependency files and three CI env: blocks. No migration, no data, no flag.
• Blast radius: every backend and worker container. Substitutive, not additive: the same torch version, a different build of it.
• Alternative considered: keeping the stack and documenting it (Alternative 1) — rejected because docling demonstrably does not need it.