fix production issues

docs/ops/canary-rollback.md

@@ -0,0 +1,47 @@
+# Canary deploy & rollback (Phase 3)
+
+**Status:** Ops runbook — evidence required before claiming production ready  
+**Date:** 2026-08-12
+
+## Goal
+
+Ship a new API image to a small fraction of traffic, watch `/metrics` + `/ready`, and roll back within minutes if error rate or latency regresses.
+
+## Preconditions
+
+- Release image tagged and pulled (`scripts/docker-prod-deploy.sh` path)
+- Metrics scraped; alerts for 5xx and readiness exist (Phase 2)
+- Previous known-good image tag recorded
+
+## Canary procedure
+
+1. Record baseline: `p95` latency, 5xx rate, `notification_outbox_pending` for 15 minutes.
+2. Deploy new tag to **one** API replica (or Traefik weight 10% if configured).
+3. Watch 15–30 minutes under real or soak traffic (`npm run test:soak`).
+4. **Promote** only if error rate and p95 within agreed budget vs baseline.
+5. **Rollback** immediately if:
+   - `/ready` flapping
+   - 5xx rate > 2× baseline
+   - Outbox pending unbounded growth
+   - Payment/webhook failures spike
+
+## Rollback
+
+```bash
+# Set IMAGE_TAG to last known-good and redeploy
+export IMAGE_TAG=<previous-good>
+bash scripts/docker-prod-deploy.sh
+```
+
+Verify `/health`, `/ready`, and a booking smoke after rollback.
+
+## Evidence to attach
+
+| Field | Value |
+|-------|-------|
+| Date | |
+| Canary tag | |
+| Previous tag | |
+| Decision | promote / rollback |
+| Metrics summary | |
+| Operator | |