PII Classification & Encryption

Field-level PII classification, AES-256-GCM encryption at rest, HMAC search hashes, and GDPR data subject request handling -- making Trust Relay's data model self-describing for compliance auditors and bank security questionnaires.

Business Value

Regulated customers (banks, fintechs) ask "where is PII stored and how is it protected?" in every security questionnaire. Trust Relay answers this with a machine-generated pii_manifest.json -- a GDPR Art. 30 artifact listing every PII field, its classification, encryption status, retention policy, and legal basis. The PII classification system also enables automated GDPR data subject requests (access, erasure, rectification) with proper AML retention handling.

Architecture

Three-layer design: classification metadata on ORM models, transparent column encryption via SQLAlchemy TypeDecorator, and GDPR DSR endpoints backed by a person resolution index.

PII Categories

Six classification levels, three requiring encryption:

Category	Encryption	Examples
`DIRECT_IDENTIFIER`	AES-256-GCM	National ID, passport number, document number
`FINANCIAL`	AES-256-GCM	IBAN, account number
`CONTACT`	AES-256-GCM	Email, phone
`QUASI_IDENTIFIER`	RLS only	Name, DOB, address, company name
`SENSITIVE`	RLS only	PEP status, sanctions hits
`DOCUMENT`	RLS only	Uploaded ID documents (MinIO)

Categories are declared as info={"pii": PIIField(category=PIICategory.X)} on SQLAlchemy columns. The PIIRegistry scans all ORM models at runtime and builds an in-memory index of classified fields. As of 2026-06-11 the codebase carries 20 PII-classified fields across 5 tables (users, cases, person_verifications, investigation_persons, investigation_accounts).

:::warning RLS is not active in the shipped configuration The categories marked "RLS only" above rely on PostgreSQL Row Level Security for tenant isolation. RLS is currently inert because the application connects as the bootstrap PostgreSQL superuser (temporal, the database_url default in app/config.py), and superusers bypass even FORCE ROW LEVEL SECURITY. The non-superuser application role exists in scripts/create_app_role.sql (trustrelay_app) but is not wired into the runtime DATABASE_URL. Until the app connects as trustrelay_app, "RLS only" fields have no enforced at-rest isolation. See Security. :::

Encryption Architecture

EncryptedText TypeDecorator

Transparent AES-256-GCM encryption at the column level. Application code reads and writes plaintext strings -- encryption happens inside SQLAlchemy's type system.

Wire format: key_id (4B) || iv (12B) || ciphertext || tag (16B)

key_id enables key rotation without re-encrypting existing data
Random IV per encryption ensures identical values produce different ciphertext
GCM mode provides authenticated encryption (integrity + confidentiality)

KeyProvider Interface

class KeyProvider(ABC):
    def get_current_key(self) -> tuple[str, bytes]: ...  # For new encryptions
    def get_key_by_id(self, key_id: str) -> bytes: ...   # For decrypting existing data

EnvKeyProvider -- reads PII_ENCRYPTION_KEY env var (hex-encoded 32-byte AES-256 key, validated at construction). PoC/staging.
NullKeyProvider -- returns empty key material for the disabled/bypass path

A KMS-backed provider (AWS KMS / HashiCorp Vault) for production key rotation is anticipated but not yet implemented -- only EnvKeyProvider and NullKeyProvider exist in key_providers.py today. EncryptedText currently instantiates EnvKeyProvider directly rather than selecting a provider through the KeyProvider ABC.

HMAC Search Hashes

Encrypted columns can't be used in WHERE clauses. For fields that need equality lookups (email, IBAN), a parallel _hash column stores HMAC-SHA256(normalized_value, pepper):

Deterministic: same value always produces the same hash
One-way: hash can't be reversed to recover the plaintext
Pepper: separate secret from the encryption key (PII_ENCRYPTION_PEPPER)
Normalized: lowercase, trimmed before hashing

JSONB PII Encryption

JSONB columns containing PII arrays (identification, phones, emails) can't use EncryptedText because the PII is nested inside JSON structures. The jsonb_crypto service encrypts specific fields within each JSON object:

JSONB Field	PII Key Encrypted	Non-PII Preserved
`emails`	Each email string	--
`phones`	`number` key	`phone_type`, `country_prefix`
`identification`	`document_number` key	`document_type`, `issuing_country`, dates

Dev/Test Bypass

When pii_encryption_enabled=False (default), EncryptedText stores plain UTF-8 bytes -- no encryption overhead in development. Set PII_ENCRYPTION_ENABLED=true + keys in production.

PII Manifest

docs/pii_manifest.json is the GDPR Art. 30 artifact -- a machine-generated inventory of all PII fields:

cd backend && python -m app.pii.manifest

Outputs:

Summary: total PII fields, encrypted fields, tables with PII
Per-table field inventory with category, encryption status, retention, legal basis
DSR scope: which tables, JSONB paths, MinIO prefixes, and Neo4j nodes contain person data

This file is committed to the repository and serves as the compliance artifact for security questionnaires, ISO 27001 audits, and GDPR Art. 30 records of processing.

Three endpoints handle natural person rights under GDPR. All three require the compliance_manager or super_admin role (require_role(...) in app/api/data_subject.py) and run inside a get_tenant_session(user.tenant_id) scope, so a request only resolves people within the caller's tenant:

Subject Access (Art. 15)

POST /api/data-subject/access
Body: { "identifier_type": "name_dob", "first_name": "...", "last_name": "...", "date_of_birth": "..." }

Returns all PII records for the person across all cases, with retention information per case.

Erasure (Art. 17)

POST /api/data-subject/erase
Body: { "identifier_type": "name_dob", ..., "reason": "..." }

Applies AML-aware erasure rules per case:

Case Status	Action	Legal Basis
Active	Refuse	GDPR Art. 17(3)(b) -- legal obligation (active case)
Closed < 5 years (or close date unknown)	Refuse	GDPR Art. 17(3)(b) -- AML 5-year retention window
Closed > 5 years	Full delete across all four DSR-scope stores	Retention expired

There is deliberately no anonymize-in-window path: inside the retention window the obliged entity must keep regulator-demandable CDD identity and verification evidence intact, so erasure refuses with the legal basis recorded; after the window it deletes across Postgres, MinIO identity documents, the goAML draft JSONB, and the Neo4j person nodes, with each store's outcome reported individually on stores.

Known residuals (ADR-0107): every erasure report — refuse and delete paths alike — additionally carries two retained_known_residual entries naming the stores erasure cannot reach by design: audit_events (immutable append-only trail, ADR-0064; the accepted remedy is per-subject crypto-shredding, ADR-0107, tracked follow-up) and evidence_bundles (SHA-256-hashed items under AMLR 5-year retention; rights actions are recorded as tombstone records alongside, never by mutating hashed items). A residual does not fail erasure_complete, but it is surfaced in the API response and on the DSR_ERASURE audit event (store_outcomes), so an erasure never claims a completeness it does not have. See Known Gaps -- the ADR-0107 reconciliation.

Rectification (Art. 16)

POST /api/data-subject/rectify
Body: { "identifier_type": "name_dob", ..., "corrections": {"first_name": "corrected"} }

Corrects allowed PII fields across all appearances of the person.

Person Resolution Index

The person_data_index table maps HMAC person hashes to data locations across tables. This enables cross-case person lookup without storing PII in the index itself. A person appearing as a director in 3 different company investigations can be found via a single hash lookup.

Audit Trail

Every DSR operation creates an audit_event with type DSR_ACCESS, DSR_ERASURE, or DSR_RECTIFICATION. These events are exempt from erasure (GDPR Art. 17(3)(e) -- establishment of legal claims).

Retention Policies

Context	Duration	Legal Basis
AML/KYB records	5 years from case closure	6AMLD Art. 40
Audit events	5 years	6AMLD + EU AI Act Art. 12
Person data (no active case)	1 year after last case	GDPR Art. 5(1)(e)
DSR audit events	Exempt from erasure	GDPR Art. 17(3)(e)

Proactive retention purge (ADR-0108, #384)

Erasure is not only reactive: a system-level Temporal Schedule (retention-purge, every retention_purge_interval_hours, enabled per deployment via RETENTION_PURGE_ENABLED) fires RetentionPurgeWorkflow → run_retention_purge(), which enumerates terminal cases per tenant and runs the same tested DSRService.handle_erasure for every subject whose AMLR window has expired. Fail-safe by construction: handle_erasure re-checks determine_erasure_action per case, so an in-window or active case can never be purged even if enumeration is wrong. Each tenant sweep that actually erases records writes an immutable RETENTION_PURGE audit event (ADR-0064).

The live posture is observable at GET /api/monitoring/retention (super_admin; rendered as the Retention card on /admin): flag state, Temporal schedule presence (a missing schedule is LOUD, never silent-green), the last RETENTION_PURGE audit tally, the per-category retention schedule, and an honest per-store coverage map — only the four DSR stores claim purged_by_sweep; Neo4j evidence artifacts/provenance records, MinIO evidence bundles, signal_events, and Letta passages are explicit not_yet_purged entries (tracked follow-up: case-scoped purge / assistance-memory TTL), and audit_events is retained_by_design (ADR-0107 crypto-shred pending). The coverage map never reports coverage that does not exist (claim-vs-check).

File Structure

backend/app/pii/
    categories.py           # PIICategory enum, PIIField dataclass
    encryption.py           # EncryptedText TypeDecorator, AES-256-GCM
    key_providers.py        # KeyProvider ABC, EnvKeyProvider, NullKeyProvider
    hashing.py              # HMAC-SHA256 search hash helpers
    jsonb_crypto.py         # Encrypt/decrypt PII inside JSONB arrays
    registry.py             # PIIRegistry -- scans models, builds field index
    manifest.py             # CLI: generates docs/pii_manifest.json
    person_index.py         # PersonIndexService
    dsr_service.py          # GDPR access/erase/rectify logic
    backfill.py             # Encrypt existing plaintext data

backend/app/api/
    data_subject.py         # 3 GDPR DSR endpoints

ADR

ADR-0036: PII Classification Architecture (docs/adr/ADR-0036-pii-classification-architecture.md)

Business Value​

Architecture​

PII Categories​

Encryption Architecture​

EncryptedText TypeDecorator​

KeyProvider Interface​

HMAC Search Hashes​

JSONB PII Encryption​

Dev/Test Bypass​

PII Manifest​

GDPR Data Subject Requests​

Subject Access (Art. 15)​

Erasure (Art. 17)​

Rectification (Art. 16)​

Person Resolution Index​

Audit Trail​

Retention Policies​

Proactive retention purge (ADR-0108, #384)​

File Structure​

ADR​