Skip to main content

ADR-0175: WAL archive publication is atomic, archiver health is current-state, and absence is a detected condition

Date: 2026-08-03 Status: Accepted Deciders: Adrian (project owner), Claude Opus 5, Codex (review, PR #1027) Refines: ADR-0105 (backup, restore and point-in-time recovery)

Context​

ADR-0105 defined the backup set — pg_basebackup + WAL archive + MinIO objects — and the restore runbook. It did not specify how a WAL segment becomes published, what "the archiver is healthy" means, or how anyone learns that a scheduled backup did not run. Each of those gaps produced a live defect.

1. The archive command was not idempotent, and the archiver wedged. Postgres retries archive_command until it succeeds and retains every unarchived segment meanwhile. A bare cp %p /wal_archive/%f fails when the destination already exists — which happens routinely after a restart or a partial run — so the archiver failed permanently on one segment and stopped advancing. Measured: four days with archive command failed with exit code 1 repeating, WAL accumulating, and no PITR coverage past the wedge point.

2. Archiver health was judged on a cumulative counter. pg_stat_archiver's failed_count never decreases, so a single historical failure made the shipper report an unhealthy archiver for ever, and an operator learned to ignore it.

3. Nothing asked whether a backup had recently succeeded. Every script reported on a run that fired. A run that never fires emits no log line, no failure and no signal, and is indistinguishable from a healthy system by every artifact the suite produces. Measured on the dev host: the daily job's log simply stopped on 2026-07-29 and nobody noticed for five days, because macOS cron does not run missed jobs on wake.

4. Fixing (1) created a false FRESH in the fix for (3), and this is the part worth recording. The atomic archive command writes %f.tmp and renames. The WAL shipper encrypted every file it found in the spool, so an in-flight or interrupted copy was published as <seg>.tmp.gpg — a real, recent, well-formed .gpg holding truncated bytes. The freshness checker's "published artifacts only" filter accepted any *.gpg. Reproduced in review: a WAL archive containing only a fresh .tmp.gpg reported FRESH and exited 0.

Both changes were correct. Landing them together was the defect: one introduced a new temporary-file shape, the other filtered against the temporary shapes that existed when it was written.

Decision​

1. Publication is atomic, everywhere, and the temp shape is excluded by every consumer.

archive_command becomes test ! -f /wal_archive/%f && cp %p /wal_archive/%f.tmp && mv /wal_archive/%f.tmp /wal_archive/%f. The test ! makes a re-archive a no-op rather than a failure; rename(2) on the same filesystem makes a reader see either the whole segment or nothing.

Because a temp shape now exists, every consumer of that directory must exclude it: the shipper skips *.tmp in the spool (root cause), and the freshness checker rejects *.tmp.gpg and *.partial (defence in depth, because an archive populated before this ADR still contains them). The same rule already applies to the logical/base/MinIO directories, whose writers publish <stamp>.incomplete and .staging-<stamp>.

The general form: an atomic-publish scheme is a contract between a writer and all of its readers. Adding a temp shape without updating every reader converts a partial write into a value that reads as complete.

2. Archiver health is a current-state predicate, not a cumulative one. Healthy means last_failed_time is not later than last_archived_time. A historical failure that has since been overtaken is history, not a fault. An unreachable cluster is exit 1 — "could not check" is not "healthy".

3. Absence is a detected condition, on an independent schedule. check_backup_freshness.sh reads the artifacts on disk rather than the logs, because a log records intentions that ran and a directory records outcomes that landed. It answers with three states — 0 fresh, 1 stale, 2 cannot assess — and never reports fresh for a directory it could not read.

It runs as its own cron line, deliberately not as a step inside backup_all.sh: a check that runs as part of the job cannot detect the job not running, which is the entire failure mode. Every documented cron line is prefixed mkdir -p backups/logs because the shell opens the redirect before starting the command, and on a fresh host the repo-local log directory does not exist — the redirect fails, the command never runs, and the freshness checker would go missing through its own installation instructions.

4. The PITR anchor is checked. WAL segments replay from a base backup, so an archive without one restores nothing. Thresholds: logical 36 h, MinIO 36 h, WAL 6 h, base backup 192 h (8 days — one day of slack on a 7-day cadence, deliberately not the 336 h "one missed week" implies).

Consequences​

Positive​

  • The archiver cannot wedge on a re-archive, and a partially copied segment can never be read as a complete one.
  • A backup that silently stops running is detected within a day instead of never.
  • "Could not look" is a distinct answer from "it is late", so a permissions or configuration fault is not investigated as a missed backup.

Negative​

  • Four thresholds are judgement calls tuned to a laptop that sleeps. On a server that is always up they are far too loose, and nothing detects that mismatch — they will need revisiting when a deployment target exists.
  • The freshness checker sources backup_common.sh for canonical configuration, so a syntax error there now breaks the checker as well as the writers. Accepted: the alternative was re-deriving the backup root independently, which is what made it read the wrong store in the first place.
  • mkdir -p in a cron line is inelegant, and it is the only place that reliably runs before the redirect.

Neutral​

  • No schema change, no application code touched; this is the backup toolchain and its runbook.
  • ADR-0105 stays Accepted. Its backup set and restore procedure are unchanged; this ADR fills in the publication, health and absence-detection semantics it left unspecified.

Alternatives Considered​

Alternative 1: cp -n (no-clobber) instead of test ! -f … && cp … && mv​

Shorter, and it makes a re-archive a no-op.

Why rejected: cp -n exits 0 when it skips, which is right, but it still writes the destination path directly, so a reader can observe a partially written segment. Atomicity and idempotence are separate properties and this only buys one.

Alternative 2: let the freshness check run as the last step of backup_all.sh​

No extra cron line, and it reports immediately after the work.

Why rejected: it cannot detect the failure it exists for. If the daily job does not fire, a check inside it does not fire either. Measured — that is exactly how five nights were lost.

Alternative 3: alert on pg_stat_archiver.failed_count > 0​

The simplest health predicate, and it was the shipped one.

Why rejected: the counter is cumulative, so one historical failure pins the alarm on permanently. An alarm that is always on is an alarm that is always ignored, which is worse than no alarm because it is counted as a control.

Decision context:

  • Latency: none meaningful. One extra test/mv per segment; the checker is a handful of find/stat calls.
  • Dependency surface: none. POSIX shell, find, stat, the existing gpg path.
  • Debuggability: the checker names the class, the directory and the age, and distinguishes cannot-assess from stale — so an operator is sent to the right problem rather than to a missing backup that is really a permissions error.
  • Reversibility: minutes. Compose-level command string, two shell scripts and a runbook block; no data migration and nothing persisted.
  • Blast radius: WAL publication for every deployment, the backup health signal, and one new cron line. Restore procedure unchanged.
  • Alternative considered: cp -n — rejected, it delivers idempotence without atomicity, and a reader can still see a half-written segment.