root
8fc88ffc14
Build & Push / Pipeline Tests (push) Failing after 59s
Build & Push / Build & Push Docker Image (push) Has been skipped
Test / Type Check (all packages) (push) Failing after 51s
Test / API Unit Tests (push) Has been skipped
Test / Homepage Unit Tests (push) Has been skipped
Test / Carplace Unit Tests (push) Has been skipped
Test / Admin Unit Tests (push) Has been skipped
Test / Dashboard Unit Tests (push) Has been skipped
Test / API Integration Tests (push) Has been skipped
1.3 KiB
1.3 KiB
Canary deploy & rollback (Phase 3)
Status: Ops runbook — evidence required before claiming production ready
Date: 2026-08-12
Goal
Ship a new API image to a small fraction of traffic, watch /metrics + /ready, and roll back within minutes if error rate or latency regresses.
Preconditions
- Release image tagged and pulled (
scripts/docker-prod-deploy.shpath) - Metrics scraped; alerts for 5xx and readiness exist (Phase 2)
- Previous known-good image tag recorded
Canary procedure
- Record baseline:
p95latency, 5xx rate,notification_outbox_pendingfor 15 minutes. - Deploy new tag to one API replica (or Traefik weight 10% if configured).
- Watch 15–30 minutes under real or soak traffic (
npm run test:soak). - Promote only if error rate and p95 within agreed budget vs baseline.
- Rollback immediately if:
/readyflapping- 5xx rate > 2× baseline
- Outbox pending unbounded growth
- Payment/webhook failures spike
Rollback
# Set IMAGE_TAG to last known-good and redeploy
export IMAGE_TAG=<previous-good>
bash scripts/docker-prod-deploy.sh
Verify /health, /ready, and a booking smoke after rollback.
Evidence to attach
| Field | Value |
|---|---|
| Date | |
| Canary tag | |
| Previous tag | |
| Decision | promote / rollback |
| Metrics summary | |
| Operator |