Skip to main content

ADR-0142: Crypto-shred key architecture + hash-over-ciphertext correction

Date: 2026-07-25 Status: Accepted (implemented behind crypto_shred_enabled, #550 Tasks 1-8; see docs/superpowers/specs/2026-07-25-crypto-shred-design.md) Deciders: Adrian (Soft4U BV), Claude (Opus 4.8) Refines: ADR-0107 (crypto-shredding reconciles GDPR erasure with the immutable audit trail — the decision; this ADR pins the architecture and corrects one load-bearing assumption in it) Extends / depends on: ADR-0064 (immutable audit_events — trigger/REVOKE/FK), ADR-0109 (append-only hash chain), ADR-0106 (EncryptedText/EncryptedJSONB PII encryption), ADR-0036 (PII classification + key-id-tagged AES-256-GCM), ADR-0105 (backup/restore — key durability), ADR-0139/0140 (retention clock + holds — the erase gate), ADR-0067 (fail-closed / never-suppress) Issue: #550 (AMLR epic #528, Wave 4)

Decision context:

  • Latency: one symmetric encrypt of the PII leaves of a PII-bearing audit event on the write path (microseconds); one DEK unwrap on read of a non-shredded subject; a shred is a single small-row destroy. Negligible.
  • Dependency surface: no new runtime dependency — reuses the app/pii/ AES-256-GCM primitives, the key-id-tagged wire format and the KeyProvider seam. One new DB table (the keystore) and one new dark-launch flag.
  • Debuggability: an erased subject's audit details decrypt as "key shredded" — an expected, logged state, not an error. The row, its type, timestamp, and hash-chain position are all intact and verifiable.
  • Reversibility: the mechanism is reversible (flag off → plaintext) until the first key is shredded; a shredded subject is intentionally irreversible (that is the erasure). Dark-launched crypto_shred_enabled=False; flag-off is byte-identical to today.
  • Blast radius: additive. Flag-off changes nothing. Flag-on changes only the newly written audit/evidence PII representation and adds a shred step to the DELETE branch of erasure. ADR-0064's immutability is untouched (no UPDATE/DELETE of an audit row, ever).
  • Alternative considered: keep 0107's pure per-subject key, and 0107's assumption that the chain already covers ciphertext — both rejected below (they do not survive contact with the shipped code).

Context​

ADR-0107 (2026-07-12) decided crypto-shredding as the resolution to GDPR Art. 17 erasure vs AMLR Art. 77 retention over PII embedded in the immutable stores. It named the decision and left the per-subject key namespace, the encrypt-on-write, and the shred-on-erase as a tracked follow-up. #550 is that follow-up. Reading the shipped code to design the implementation surfaced two things ADR-0107 could not have known, because the relevant code landed after it:

  1. ADR-0107's central hash claim is false against the shipped chain. 0107 §Consequences(Positive) asserts: "the chain covers ciphertext, so shredding a key does not break chain verification." The ADR-0109 hash chain (#286) landed after 0107 and hashes over plaintext: AuditService.log_event writes details=<plaintext dict> (audit_service.py:104,118) and computes entry_hash = compute_entry_hash(prev_hash, row) (:110), whose canonical_row includes details in HASHED_FIELDS (audit_hash.py:37-48) and serialises the plaintext (:82-91). audit_events.details is a plain JSONB column — no EncryptedText. Evidence bundles likewise hash plaintext (data_hash=hash_data(<plaintext>), evidence_bundle_service.py:161-162,175; hash_data, trustrelay_models/evidence_bundle.py:77-85) and store plaintext inline. The case-pack Merkle pack_hash (case_pack_service.py:488-489) is over rendered bytes that embed plaintext PII. So there is no ciphertext to shred today, and naively encrypting after-the-fact would change the hash. Crypto-shred as 0107 literally describes it is not yet possible.

  2. A pure per-subject key cannot honour the per-case erasure decision. Erasure is already decided per (subject, case): handle_erasure iterates PersonDataIndex per case and calls determine_erasure_action per case; a #549 legal hold flips DELETE→REFUSE per case (dsr_service.py:652-703). The module's invariant (:608-613): a subject in a held case and an expired case is retained in the former, erased in the latter. One per-subject key shredded across all cases would erase them from the held case too — a retention breach.

The fix must (a) make every hash compute over ciphertext so a shred never breaks the chain, (b) key the ciphertext at the granularity the erasure decision is actually made, and (c) never weaken ADR-0064 immutability, the ADR-0109 chain, or the ADR-0139/0140 fail-closed retention gate.

Decision​

1. Hash over ciphertext, encrypted at the write path — the correction to ADR-0107. When crypto_shred_enabled is on, the PII sub-values of audit_events.details (and evidence-bundle data) are replaced with key-id-tagged AES-256-GCM ciphertext before compute_entry_hash / hash_data runs. The hash is therefore computed over the ciphertext; destroying the key leaves the stored details bytes, the entry_hash, and the prev_hash links unchanged, so verify_chain still passes and the shredded PII simply fails to decrypt. Non-PII metadata stays plaintext and queryable. A field-level PII classifier (PIIRegistry-introspection, never a hand list — the manifest.py:_encrypting_columns discipline) decides which leaves are PII; free-text/narrative keys are fail-closed treated as PII-bearing. New event types SHOULD reference a person_hash, not embed a name (ADR-0107 §Alt-3, reference-don't-embed), shrinking the classifier surface.

2. Per-(subject, case) shreddable DEK, wrapped by a per-tenant KEK (refines 0107's "per-subject"). The shreddable unit is a per-(subject, case) Data-Encryption Key — the exact granularity at which handle_erasure already grants/refuses erasure, so the shred inherits that gate and no new retention logic can drift from determine_erasure_action. subject_ref = the existing non-reversible person_hash (hashing.py); case_id nullable (NULL = subject-wide, for case-less DSR events). DEKs are wrapped under a per-tenant KEK (today the EnvKeyProvider key; a KMS/HSM later per ADR-0105) and stored wrapped in a new RLS keystore table crypto_shred_keys (both tenant_isolation + admin_bypass policies — the #561 gap, from the start). The KEK never decrypts a PII value directly; it only unwraps DEKs.

3. Destroy = overwrite + delete the one wrapped-DEK row. The wrapped DEK is the only copy; the KEK alone cannot reconstruct it; therefore destroying it renders every ciphertext under that DEK — including the untouched immutable audit_events rows — permanently undecryptable. "Deleted" = "the plaintext can no longer be produced by anyone, including us" (ADR-0107). The crypto_shred_keys table is deliberately the one store where DELETE is required, the mirror of audit_events where it is forbidden.

4. One canonical shred event, immutable and audited. The shred writes a single crypto_shred audit_events row (subject_ref/case_id/scope/key_id/basis/dsr_id/shredded_at/ verification) — itself immutable and hash-chained, carrying no plaintext PII, so it is the durable proof-of-erasure (EU AI Act Art. 12 / AMLR Art. 77(1) assessment record). The DSR console, the evidence surface, the case-pack masked render, and GET /monitoring/retention all read this one event — never a per-surface re-derivation.

5. Shred on the DELETE branch, fail-closed. CryptoShredService.shred runs inside the DELETE branch of handle_erasure (so the #548/#549 retention clock + holds gate it — a held case never shreds), destroys the DEK, verifies the state (a previously-decryptable ciphertext now fails), writes the event, and returns a store result that replaces today's retained_known_residual with crypto_shredded. A failed destroy or a still-decryptable ciphertext → failed → erasure_complete=False. A unit with no DEK (pre-cutover plaintext) → honest retained_known_residual, never a silent "erased".

6. Scope boundary. #550 covers the immutable stores where deletion is impossible — audit_events and evidence bundles. investigation_persons / person_verifications are row-deletable (handle_erasure:721-746) and stay on ADR-0106 EncryptedText under the platform key; re-keying them per-subject is out of scope.

Full rollout (expand-contract, dark-launched, the mandatory throwaway-DB end-to-end proof per the ADR-0106/#309 lesson), the keystore schema, the field classifier, and the task breakdown are specified in docs/superpowers/specs/2026-07-25-crypto-shred-design.md.

Consequences​

Positive​

  • Both obligations satisfied at once, provably: the audit trail stays immutable + hash-verifiable, and an erased subject's PII becomes cryptographically unrecoverable. "Prove this subject is gone" is answered by the crypto_shred event + a row that no longer decrypts + a chain that still verifies.
  • The per-(subject, case) granularity means the shred reuses the existing per-case erasure gate verbatim — no second retention decision to drift from determine_erasure_action / the ADR-0140 hold logic.
  • Reuses all existing crypto primitives and the immutable-table invariants unchanged; the only wire-format change (widening the key-id) affects only newly written ciphertext.

Negative​

  • The keystore becomes as critical as the audit trail. Losing a DEK before retention expires is an early-erasure data-loss event; a backup carrying a wrapped DEK can resurrect a shredded subject on restore (the ADR-0105 tension — OPEN, must be resolved before Phase 2 goes live: KMS-authoritative delete vs short keystore-backup retention + shred-replay).
  • Historical plaintext audit PII is a known residual. Rows written before the encrypt-at-write cutover hash over plaintext and cannot be retro-encrypted (immutable). handle_erasure keeps disclosing them as retained_known_residual; they age out with the 5-year horizon.
  • Erasure by cryptographic assumption (AES-256-GCM unbroken), not physical overwrite — standard for this threat model, weaker than deletion, documented not hidden (ADR-0107 §Negative).
  • The shared HMAC pepper makes person_hash a confirmation oracle for a known candidate identity even after a shred — mitigated by treating person_hash as a protected non-reversible id (pepper-as-secret, GDPR pseudonymisation); OPEN.

Neutral​

  • Non-PII audit metadata stays plaintext and queryable — dashboards/monitoring unaffected.
  • Flag-off is byte-identical to today; the DSR report shape is unchanged (a residual entry flips to a shred entry only when the flag is on and the row was written encrypted).

Alternatives considered​

Alternative 1: keep ADR-0107's pure per-subject key​

Rejected — it cannot honour the per-case erasure decision the code already makes (a held case would be erased when the subject is shredded elsewhere). Per-(subject, case) matches the existing gate exactly.

Alternative 2: hash a redacted tombstone from the start (not ciphertext)​

Rejected as the default — it strips the audit trail's evidentiary value during the retention window (a regulator reading the trail needs the real content). Hashing ciphertext keeps full value during retention and yields an unrecoverable-but-hash-stable row after the shred. Tombstone is retained only for event types that must never carry PII (reference-don't-embed).

Alternative 3: make audit_events mutable and redact PII in place on erasure​

Rejected (as ADR-0107 §Alt-2 already did) — it destroys the immutability AMLR/EU AI Act Art. 12 require; a trail that can be redacted on request can be redacted by an attacker.

Implementation status​

Implemented (all eight tasks, #550), dark-launched behind crypto_shred_enabled (default false). Flag-off is byte-identical to pre-#550; flipping the flag on remains Calibration-Review-gated. The eight-task breakdown in the design spec is complete:

  1. Keystore + key provider — Alembic crypto_shred_keys (RLS, both tenant_isolation + admin_bypass); SubjectKeyProvider (get-or-create DEK, KEK-wrap/unwrap under the distinct ShredKekProvider / CRYPTO_SHRED_KEK, shred); widened key-id format (app/pii/encryption.py).
  2. Hash-preservation proof harness (tests/test_crypto_shred_hash_preservation.py) — the load-bearing invariant, TDD-first.
  3. PII field classifier + encrypt/decrypt (app/pii/details_crypto.py) — PIIRegistry-introspected, fail-closed; decrypt tolerates pre-cutover plaintext.
  4. Encrypt-at-write wired into audit_service.log_event (pre-hash) + build_bundle, flag-gated.
  5. CryptoShredRecord + CryptoShredService.shred (app/services/crypto_shred_service.py) — destroy + verify-state + the immutable crypto_shred event; plus the shred-replay journal (crypto_shred_journal, Alembic 098) — the §6.1 option-(a) interim backup-resurrection defence.
  6. Erase-path wiring — handle_erasure residual→shred on the DELETE branch; run_retention_purge inherits; fail-closed store results; #549 hold → no shred; replay_shred_journal + the restore runbook.
  7. Surface consumers — DSR console crypto_shredded store entry, evidence surface, case-pack masked [erased — crypto-shredded] render, GET /monitoring/retention crypto_shred_enforced flip.
  8. End-to-end lifecycle proof + docs (tests/test_crypto_shred_lifecycle_e2e.py) — the §5 four-step proof on a real testcontainer DB (seed plaintext → flip on → ciphertext → shred → chain still verifies while PII no longer decrypts → case-pack masks + re-seals); regenerated docs/pii_manifest.json; the Docusaurus Privacy & Data-Protection + Known Gaps pages; and this Status flip.

The two OPEN items in §Negative are handled as designed, not silently: the backup-resurrection tension (§6.1) ships the interim option-(a) shred-replay journal + the export/restore/replay runbook (Known Gaps → "Crypto-shred restore procedure", docusaurus/trust-relay/docs/architecture/known-gaps.md), with the durable separate-store / KMS answer (option (b)) tracked as production hardening; the person_hash confirmation-oracle (§6.3) rests on treating person_hash as a protected non-reversible id (pepper-as-secret / GDPR pseudonymisation), consistent with its use everywhere else. Per the s4u-ADR convention, going forward only the Status line of this Accepted ADR changes; this section is the record of the implementation.

ADR-0107's residual-disclosure language and its "Implementation status / pending" note are updated to point here (the retained_known_residual disclosures remain the honest flag-off / pre-cutover state; a fully-erased case's audit_events entry flips to crypto_shredded when the flag is on).