root
8fc88ffc14
Build & Push / Pipeline Tests (push) Failing after 59s
Build & Push / Build & Push Docker Image (push) Has been skipped
Test / Type Check (all packages) (push) Failing after 51s
Test / API Unit Tests (push) Has been skipped
Test / Homepage Unit Tests (push) Has been skipped
Test / Carplace Unit Tests (push) Has been skipped
Test / Admin Unit Tests (push) Has been skipped
Test / Dashboard Unit Tests (push) Has been skipped
Test / API Integration Tests (push) Has been skipped
48 lines
1.3 KiB
Markdown
48 lines
1.3 KiB
Markdown
# Canary deploy & rollback (Phase 3)
|
||
|
||
**Status:** Ops runbook — evidence required before claiming production ready
|
||
**Date:** 2026-08-12
|
||
|
||
## Goal
|
||
|
||
Ship a new API image to a small fraction of traffic, watch `/metrics` + `/ready`, and roll back within minutes if error rate or latency regresses.
|
||
|
||
## Preconditions
|
||
|
||
- Release image tagged and pulled (`scripts/docker-prod-deploy.sh` path)
|
||
- Metrics scraped; alerts for 5xx and readiness exist (Phase 2)
|
||
- Previous known-good image tag recorded
|
||
|
||
## Canary procedure
|
||
|
||
1. Record baseline: `p95` latency, 5xx rate, `notification_outbox_pending` for 15 minutes.
|
||
2. Deploy new tag to **one** API replica (or Traefik weight 10% if configured).
|
||
3. Watch 15–30 minutes under real or soak traffic (`npm run test:soak`).
|
||
4. **Promote** only if error rate and p95 within agreed budget vs baseline.
|
||
5. **Rollback** immediately if:
|
||
- `/ready` flapping
|
||
- 5xx rate > 2× baseline
|
||
- Outbox pending unbounded growth
|
||
- Payment/webhook failures spike
|
||
|
||
## Rollback
|
||
|
||
```bash
|
||
# Set IMAGE_TAG to last known-good and redeploy
|
||
export IMAGE_TAG=<previous-good>
|
||
bash scripts/docker-prod-deploy.sh
|
||
```
|
||
|
||
Verify `/health`, `/ready`, and a booking smoke after rollback.
|
||
|
||
## Evidence to attach
|
||
|
||
| Field | Value |
|
||
|-------|-------|
|
||
| Date | |
|
||
| Canary tag | |
|
||
| Previous tag | |
|
||
| Decision | promote / rollback |
|
||
| Metrics summary | |
|
||
| Operator | |
|