fix(entrypoint): robust auto-restore - deterministic, loud, never silently skips #141

Merged

fastie81 merged 1 commit from fix/auto-restore-robust into main

2026-09-06 21:36:42 +12:00

fastie81 commented

2026-09-06 21:19:04 +12:00

Owner

Problem (prod incident)

On first PG18 boot the auto-restore silently skipped, so the app migrated a fresh EMPTY DB (booted with 0 members). Data was recovered manually from the pg15 pre-upgrade dump. The bug: the restore scan was a fragile chain (glob+sed+-lt+guards) with no logging and a silent-skip path — if PG_MAJOR came up empty or the scan mis-filtered, PREV_DUMP was empty and the block printed nothing, then booted on empty.

Fix (verified against real Postgres)

Rewrite of the auto-restore block to be robust and loud:

Resolve + LOG server PG major up front (always, for dump+restore+migrate).
Deterministic dump pick: only numeric-major files, only majors STRICTLY lower than running, only NON-EMPTY files (ignores 0-byte artifacts from pg_dump failures), and among qualified picks the highest prior major.
Log every branch loudly: qualify / skip (each reason) / restored / not-empty / none-found.
CRITICAL fail-hard guard: if a prior-major dump exists but the server major or DB-state cannot be determined, exit 1 — never boot on a possibly-empty DB silently.

Verification (real Postgres, isolated)

Seed source (3 clubs × 25 = 75 members, 10 users) → pg15 dump → fresh target → robust logic restored successfully → target = 75/10/3.
0-byte artifact rejection: picks the real non-empty dump, skips 0-byte files.
No-dump path: correctly logs "nothing to auto-restore", no wrong restore.
bash -n OK.

Deploy plan (per user instruction)

Merge → deploy to TEST → verify auto-restore actually fires+logs there (test is currently empty PG18, a perfect canary) → only then tag v3.4.9 to prod. Prod is healthy and data-verified now; this protects the next upgrade from the silent-skip bug.

## Problem (prod incident) On first PG18 boot the auto-restore **silently skipped**, so the app migrated a fresh EMPTY DB (booted with 0 members). Data was recovered manually from the pg15 pre-upgrade dump. The bug: the restore scan was a fragile chain (glob+sed+`-lt`+guards) with **no logging and a silent-skip path** — if `PG_MAJOR` came up empty or the scan mis-filtered, `PREV_DUMP` was empty and the block printed nothing, then booted on empty. ## Fix (verified against real Postgres) Rewrite of the auto-restore block to be robust and loud: - **Resolve + LOG server PG major up front** (always, for dump+restore+migrate). - **Deterministic dump pick**: only numeric-major files, only majors STRICTLY lower than running, only **NON-EMPTY** files (ignores 0-byte artifacts from `pg_dump` failures), and among qualified picks the **highest prior major**. - **Log every branch loudly**: qualify / skip (each reason) / restored / not-empty / none-found. - **CRITICAL fail-hard guard**: if a prior-major dump exists but the server major or DB-state cannot be determined, `exit 1` — never boot on a possibly-empty DB silently. ## Verification (real Postgres, isolated) - Seed source (3 clubs × 25 = 75 members, 10 users) → `pg15` dump → fresh target → robust logic **restored successfully** → target = **75/10/3**. - **0-byte artifact rejection**: picks the real non-empty dump, skips 0-byte files. - **No-dump path**: correctly logs "nothing to auto-restore", no wrong restore. - `bash -n` OK. ## Deploy plan (per user instruction) Merge → deploy to **TEST** → verify auto-restore actually fires+logs there (test is currently empty PG18, a perfect canary) → only then tag `v3.4.9` to prod. Prod is healthy and data-verified now; this protects the next upgrade from the silent-skip bug.

fastie81 added 1 commit

2026-09-06 21:19:06 +12:00

fix(entrypoint): robust auto-restore - deterministic, loud, never silently skips

CI / test (push) Successful in 2m24s

Details

CI / lint-frontend (push) Successful in 56s

Details

CI / build (push) Has been skipped

Details

CI / deploy (push) Has been skipped

Details

65acbc415c

Prod incident: on first PG18 boot the auto-restore silently skipped, so the app
migrated a fresh EMPTY DB (booted with 0 members). Data was recovered from the
pg15 pre-upgrade dump.

Root cause: the restore scan was a fragile chain (glob+sed+-lt+guards) with no
logging and a silent-skip path when PG_MAJOR came up empty. This makes it robust:
- Resolve+LOG server PG major up front.
- Deterministic dump pick: only numeric-major files, only majors STRICTLY lower,
  only NON-EMPTY files (ignores 0-byte artifacts from pg_dump failures), and
  among qualified picks the HIGHEST prior major.
- Log EVERY branch loudly (qualify/skip/restored/not-empty/none).
- CRITICAL guard: if a prior-major dump exists but server major or DB-state
  cannot be determined, ABORT (exit 1) rather than boot on a possibly-empty DB.

Verified against real Postgres: seed->dump(pg15)->fresh target->restore success
(75/10/3), 0-byte artifact rejection, and no-dump path all correct.

fastie81 merged commit fec0175f68 into main

2026-09-06 21:36:42 +12:00