# Canary deploy & rollback (Phase 3)

**Status:** Ops runbook — evidence required before claiming production ready  
**Date:** 2026-08-12

## Goal

Ship a new API image to a small fraction of traffic, watch `/metrics` + `/ready`, and roll back within minutes if error rate or latency regresses.

## Preconditions

- Release image tagged and pulled (`scripts/docker-prod-deploy.sh` path)
- Metrics scraped; alerts for 5xx and readiness exist (Phase 2)
- Previous known-good image tag recorded

## Canary procedure

1. Record baseline: `p95` latency, 5xx rate, `notification_outbox_pending` for 15 minutes.
2. Deploy new tag to **one** API replica (or Traefik weight 10% if configured).
3. Watch 15–30 minutes under real or soak traffic (`npm run test:soak`).
4. **Promote** only if error rate and p95 within agreed budget vs baseline.
5. **Rollback** immediately if:
   - `/ready` flapping
   - 5xx rate > 2× baseline
   - Outbox pending unbounded growth
   - Payment/webhook failures spike

## Rollback

```bash
# Set IMAGE_TAG to last known-good and redeploy
export IMAGE_TAG=<previous-good>
bash scripts/docker-prod-deploy.sh
```

Verify `/health`, `/ready`, and a booking smoke after rollback.

## Evidence to attach

| Field | Value |
|-------|-------|
| Date | |
| Canary tag | |
| Previous tag | |
| Decision | promote / rollback |
| Metrics summary | |
| Operator | |