Phase 6 — Staging Validation & Production Cutover #111

New issue

Open

opened 2026-07-09 09:56:57 +12:00 by fastie81 · 0 comments

fastie81 commented

2026-07-09 09:56:57 +12:00

Owner

Issue: Phase 6 — Staging Validation & Production Cutover

Type: Operations / Deployment
Priority: High
Epic: Epic: Database-per-Club Multi-Tenancy Split & Data Migration
Status: Backlog

1. Context & Purpose

With code infrastructure, migrations, repositories, backups, and migration scripts tested and ready, we need to deploy the changes to staging for validation, and execute the final production cutover within a scheduled maintenance window.

2. Technical Specification & Checklist

2.1. Staging Deploy & Validation

Deploy the dynamic routing code, repository updates, and the ETL script to the staging environment.
Export a copy of the monolithic production database.
Run the ETL script on staging targeting this backup.
Execute E2E verification:
- Verify that staging users can log in.
- Verify that member profiles are visible and images load correctly.
- Verify that OAuth PKCE logins work.
- Run a system backup and restore cycle on staging to verify multi-database backup compatibility.
- Run pytest on staging to ensure zero regression.

2.2. Production Cutover Protocol

A scheduled maintenance window must be announced to users. The steps are:

Pre-Migration Checks:
- Ensure the PG server has sufficient disk space for temporary copy overhead (at least 2.5x current database size).
- Ensure you have shell access to the production host.
Maintenance Mode ON:
- Update the load balancer or application configuration to route all incoming traffic to a static 503 Maintenance Page.
- Wait 30 seconds for in-flight requests to drain.
- Verify there are no active write transactions on the source database:
  SELECT * FROM pg_stat_activity WHERE state != 'idle';
Backup Source Monolith:
- Run a full logical backup of the monolithic production database:
  pg_dump -h <host> -U postgres -d honbu_manager -Fc > pre_split_monolith.dump
- Store this backup safely.
Execute ETL migration:
- Run the script:
  python scripts/etl_migrate.py --source $SOURCE_DATABASE_URL --central $CENTRAL_DATABASE_URL
- Run verify mode:
  python scripts/etl_migrate.py --verify-only --source $SOURCE_DATABASE_URL --central $CENTRAL_DATABASE_URL
Switchover Configuration:
- Update the production environment variable DATABASE_URL to point to the new Central database URI (honbu_central).
- Restart the application container instances.
Smoke Tests:
- Log in using a System Admin account.
- Access the dashboard and verify stats are aggregated.
- Log in as a Club Admin and verify member listings load.
Maintenance Mode OFF:
- Disable the maintenance page and restore public routing to the app.

2.3. Post-Migration Monitoring

Monitor container metrics and database connection pool usage.
Tail application logs for any database connection errors.
Keep the old monolithic database running in read-only mode for 72 hours. Do not modify or delete it.

3. Rollback Protocol

If E2E smoke tests fail or critical data errors are reported post-migration, execute these rollback steps:

Enable Maintenance Mode:
- Re-route traffic to the static 503 Maintenance Page.
Revert Environment Configuration:
- Change the DATABASE_URL environment variable back to the monolithic database connection string.
Restart Application Containers:
- Restart application instances.
Perform Smoke Verification:
- Verify logins work and data is readable.
Disable Maintenance Mode:
- Restore public routing.
Acknowledge Data Loss Warning:
- Note: Any data created or modified in the multi-database setup during the window before rollback will be lost. This risk is minimized by performing extensive smoke tests immediately after cutover.

Part of Epic #105

# Issue: Phase 6 — Staging Validation & Production Cutover **Type:** Operations / Deployment **Priority:** High **Epic:** [Epic: Database-per-Club Multi-Tenancy Split & Data Migration](file:///mnt/f/forgejo-git/club-manager/docs/database_split_issue.md) **Status:** Backlog --- ## 1. Context & Purpose With code infrastructure, migrations, repositories, backups, and migration scripts tested and ready, we need to deploy the changes to staging for validation, and execute the final production cutover within a scheduled maintenance window. --- ## 2. Technical Specification & Checklist ### 2.1. Staging Deploy & Validation - Deploy the dynamic routing code, repository updates, and the ETL script to the staging environment. - Export a copy of the monolithic production database. - Run the ETL script on staging targeting this backup. - Execute E2E verification: - Verify that staging users can log in. - Verify that member profiles are visible and images load correctly. - Verify that OAuth PKCE logins work. - Run a system backup and restore cycle on staging to verify multi-database backup compatibility. - Run `pytest` on staging to ensure zero regression. ### 2.2. Production Cutover Protocol A scheduled maintenance window must be announced to users. The steps are: 1. **Pre-Migration Checks**: - Ensure the PG server has sufficient disk space for temporary copy overhead (at least 2.5x current database size). - Ensure you have shell access to the production host. 2. **Maintenance Mode ON**: - Update the load balancer or application configuration to route all incoming traffic to a static 503 Maintenance Page. - Wait 30 seconds for in-flight requests to drain. - Verify there are no active write transactions on the source database: `SELECT * FROM pg_stat_activity WHERE state != 'idle';` 3. **Backup Source Monolith**: - Run a full logical backup of the monolithic production database: `pg_dump -h <host> -U postgres -d honbu_manager -Fc > pre_split_monolith.dump` - Store this backup safely. 4. **Execute ETL migration**: - Run the script: `python scripts/etl_migrate.py --source $SOURCE_DATABASE_URL --central $CENTRAL_DATABASE_URL` - Run verify mode: `python scripts/etl_migrate.py --verify-only --source $SOURCE_DATABASE_URL --central $CENTRAL_DATABASE_URL` 5. **Switchover Configuration**: - Update the production environment variable `DATABASE_URL` to point to the new Central database URI (`honbu_central`). - Restart the application container instances. 6. **Smoke Tests**: - Log in using a System Admin account. - Access the dashboard and verify stats are aggregated. - Log in as a Club Admin and verify member listings load. 7. **Maintenance Mode OFF**: - Disable the maintenance page and restore public routing to the app. ### 2.3. Post-Migration Monitoring - Monitor container metrics and database connection pool usage. - Tail application logs for any database connection errors. - Keep the old monolithic database running in read-only mode for 72 hours. Do not modify or delete it. --- ## 3. Rollback Protocol If E2E smoke tests fail or critical data errors are reported post-migration, execute these rollback steps: 1. **Enable Maintenance Mode**: - Re-route traffic to the static 503 Maintenance Page. 2. **Revert Environment Configuration**: - Change the `DATABASE_URL` environment variable back to the monolithic database connection string. 3. **Restart Application Containers**: - Restart application instances. 4. **Perform Smoke Verification**: - Verify logins work and data is readable. 5. **Disable Maintenance Mode**: - Restore public routing. 6. **Acknowledge Data Loss Warning**: - Note: Any data created or modified in the multi-database setup during the window before rollback will be lost. This risk is minimized by performing extensive smoke tests immediately after cutover. --- **Part of Epic #105**