Skip to main content

ADR-0107: Crypto-shredding reconciles GDPR erasure with the immutable audit trail

Date: 2026-07-12 Status: Accepted Deciders: Adrian (Soft4U BV), Claude Opus 4.8

Decision context:

  • Latency: negligible on the write path (one symmetric encrypt of the PII fields of an audit event — microseconds). No impact on the read/verify path unless a specific subject's records are being read, which already touches the key provider.
  • Dependency surface: none new. Reuses the existing app/pii/ AES-256-GCM primitives, KeyProvider abstraction and key-id-tagged ciphertext (ADR-0036). The only new concept is a per-subject key namespace, which the existing provider can express.
  • Debuggability: an erased subject's audit rows become undecryptable by design. The row, its type, its timestamp and its hash-chain position (when #286 lands) all remain — only the PII payload is gone. A "cannot decrypt: key shredded" is an expected, logged state, not an error.
  • Reversibility: intentionally irreversible for the erased data — that is the whole point of an erasure. The mechanism is reversible (a config flip back to plaintext audit details) until the first key is shredded; after that, shredded subjects cannot be recovered. This is a property, not a risk.
  • Blast radius: additive. audit_events.details gains an encrypted representation for PII fields; existing non-PII details are unaffected. No schema-breaking change to the immutable table (the trigger/REVOKE/FK of ADR-0064 are untouched — we never UPDATE or DELETE a row).
  • Alternative considered: legal-basis retention exemption (keep the PII, cite AML) — rejected below; it does not satisfy an erasure that is due.

Context​

Two obligations point in opposite directions on the same table.

AMLR / EU AI Act Art. 12 require an audit trail that is retained for 5 years and cannot be altered. ADR-0064 enforces this at the database: a guard trigger, a REVOKE UPDATE, DELETE on audit_events, and an FK RESTRICT. Rows are append-only and immutable. The regulator-ready case pack (ADR-0069) is built from this table, and its whole value is that history cannot be rewritten.

GDPR Art. 17 gives a data subject the right to erasure. After the AMLR retention window expires (or where erasure is otherwise due), the subject's personal data must be deleted.

M1-W6 (finding D5) surfaced the collision concretely: audit_events.details is a JSONB column, unencrypted, and it contains personal data — names, and the substance of findings about a person. A completed Art. 17 erasure today deletes the person's rows from investigation_persons and person_verifications, and (with the rest of W6) from MinIO, the goAML drafts and the graph — but their PII remains, in plaintext, in the immutable audit trail that is specifically designed so that nothing can remove it.

The two controls are each individually correct. Their intersection is a defect: an erasure that reports success while the subject's data persists, in the one store built to be un-erasable. That is this programme's recurring defect class — a control (erasure) that reports completion without achieving its state — occurring at the seam between two other controls.

You cannot resolve this by weakening either side. Making audit_events mutable to allow erasure destroys the immutability AMLR requires. Refusing erasure to preserve immutability breaches Art. 17. The record must stay and the PII must go.

Decision​

Store the PII fields of audit_events.details encrypted under a per-subject key, and perform erasure by destroying that key — not by deleting the row.

Crypto-shredding. The audit row is never updated or deleted, so ADR-0064's immutability is untouched. What changes is that the row's personal-data payload is ciphertext, and the key that decrypts it is scoped to a single data subject. When that subject is erased, their key is shredded from the KeyProvider. The row remains — auditable in shape, timestamp, type and (per #286) hash-chain position — but its PII is cryptographically unrecoverable. "Deleted" is redefined, correctly and defensibly, as "the plaintext can no longer be produced by anyone, including us."

Concretely:

  1. Per-subject key. Extend the KeyProvider (ADR-0036) with a per-subject key namespace (subject:{subject_id}). The existing key-id-tagged ciphertext format already carries the key id, so a mix of platform-key and subject-key ciphertexts coexists without ambiguity.
  2. Encrypt PII details on write. When an audit event carries personal data, its PII fields (not the whole details blob — non-PII metadata stays queryable) are encrypted under the acting subject's key before the row is written. A field-level PIIRegistry-style annotation decides what is PII, so this is not a hand-maintained list of fields.
  3. Erase by shredding, in the same DSR transaction. When DSRService performs a DELETE action for a subject, after erasing the four data stores (W6) it destroys subject:{subject_id} from the key store. The audit rows are left in place. The DSR result records "audit PII crypto-shredded" as one of the erased surfaces, so the erasure report is complete and honest.
  4. Verification. A shredded subject's audit details must fail to decrypt, and the DSR completeness test asserts exactly that — the same "assert the state, not the intent" discipline the rest of the programme uses.

The key store's own durability now carries a retention obligation: a subject key must survive as long as the audit trail needs to be readable (5 years) and no longer than the subject's erasure allows. Key backup/retention is folded into the backup design (ADR-0105).

Consequences​

Positive​

  • Both obligations are satisfied simultaneously: the audit trail stays immutable and 5-year retained; an erased subject's PII becomes unrecoverable. Neither control is weakened.
  • Erasure is now provable. "Prove this subject's data is gone" is answered by "their key is shredded; here is the row that no longer decrypts" — evidence, not a claim.
  • Reuses existing crypto primitives; no new dependency, no schema-breaking change to the immutable table.
  • Extends cleanly to the hash-chain work (#286): the chain covers ciphertext, so shredding a key does not break chain verification — the row and its hash are unchanged.

Negative​

  • The key store becomes a second thing that must be as durable and as backed-up as the audit trail itself. Lose a subject's key before their retention expires and you have effectively erased them early — a data-loss event dressed as a shred. This shifts risk from the database to key management, which must be treated with the same seriousness (ADR-0105 backup scope now includes subject keys).
  • Crypto-shredding is erasure by cryptographic assumption, not by overwrite. It relies on AES-256-GCM remaining unbroken. For the retention horizons and threat model here this is standard and accepted, but it is a weaker guarantee than physical deletion, and a regulator who insists on overwrite semantics would not be satisfied by it. Documented, not hidden.
  • Adds encryption to the audit write path for PII-bearing events. Cheap, but non-zero, and it means the audit trail is now only as readable as the key store is available.
  • Migration: existing plaintext audit_events.details rows are already immutable, so they cannot be retro-encrypted in place (no UPDATE permitted). Historical PII in the audit trail is a pre-existing exposure this ADR does not close for rows already written; it can only be addressed by the retention purge (D4) reaching those rows, or by an out-of-band superuser re-write that ADR-0064 deliberately makes hard. This is a real residual and is tracked as a follow-up, not waved away.

Neutral​

  • Non-PII audit metadata stays plaintext and queryable — dashboards and monitoring that read event types, timestamps and counts are unaffected.
  • The DSR erasure report gains one more line item ("audit PII shredded"); no change to its shape.

Alternatives Considered​

Alternative 1: Legal-basis retention exemption — keep the PII, cite AML​

  • GDPR Art. 17(3)(b) allows retention where processing is necessary for a legal obligation. Keep the audit PII in plaintext and refuse erasure of it on AML grounds.
  • Why rejected: it only covers PII within the AML retention window and for the AML purpose. Once retention expires, or for a subject whose data was never AML-relevant, the exemption does not apply and the erasure is genuinely due — at which point plaintext audit PII is an unremedied breach. It answers the easy half of the problem and leaves the hard half (post-retention erasure) exactly where it was. It also conflates "we may keep it" with "we cannot remove it," which is the confusion that produced the defect.

Alternative 2: Make audit_events mutable and redact PII in place on erasure​

  • Drop ADR-0064's REVOKE/trigger, and on erasure UPDATE the affected rows to null out PII.
  • Why rejected: it destroys the property AMLR and EU AI Act Art. 12 require — that the audit trail cannot be altered. A trail that can be redacted on request can be redacted by an attacker or an insider, and its evidentiary value collapses. This trades a GDPR problem for a larger AML/AI-Act one. The whole point of ADR-0064 is that no one can change a row; an erasure path that changes rows is that door, held open.

Alternative 3: Never put PII in audit_events.details at all​

  • Store only foreign keys / pseudonymous references in the audit trail; keep the PII solely in the erasable person tables, and resolve references at read time.
  • Why rejected (partially adopted): it is the right principle going forward and should be applied to new event types — an audit event should reference a person, not embed them. But (a) it does not address the PII already embedded in historical rows, and (b) some audit events must record the substance of a finding about a person for the trail to be meaningful to a regulator ("flagged as PEP because X"), and that substance is itself PII. So pseudonymisation reduces the surface but cannot eliminate it, and crypto-shredding is still needed for the irreducible remainder. We adopt the reference-don't-embed principle for new events and crypto-shredding for the PII that must be embedded.

Implementation status​

This ADR is the decision. The per-subject key namespace, the encrypt-on-write for PII audit details, and the shred-on-erase step are a tracked follow-up (they touch the key provider and the audit write path and warrant their own reviewed change). M1-W6 delivers the four-store erasure and the retention purge; the audit crypto-shred lands next, against this ADR. Until it does, DSRService records audit-trail PII as a known residual in its erasure report rather than claiming a completeness it does not yet have — an honest gap, not a silent one.

Update 2026-07-25 (ADR-0142, #550) — the tracked follow-up above is now implemented, dark-launched behind crypto_shred_enabled (default false). The follow-up was designed and built as ADR-0142 (Crypto-shred key architecture + hash-over-ciphertext correction — refines this ADR), which also corrects one assumption here: the §Positive claim that "the chain covers ciphertext" was false against the shipped ADR-0109 chain (it hashes over plaintext, having landed after this ADR), so ADR-0142 encrypts the PII sub-values before the row is hashed. The "per-subject key" of the Decision became a per-(subject, case) DEK (the granularity the erasure decision is already made at) wrapped by a distinct per-tenant KEK. Flag-on, a fully-erased case's audit_events residual flips from retained_known_residual to crypto_shredded (carrying the shred_id); the ADR-0109 chain still verifies while the PII no longer decrypts. What stays a residual: historical plaintext written before the encrypt-at-write cutover is immutable and cannot be retro-encrypted, so those specific rows keep disclosing retained_known_residual. See Privacy & Data-Protection → Crypto-shred and the Known Gaps restore runbook. Flag-off is byte-identical to this ADR's pre-#550 disclosure state.