fix production issues

2026-08-12 16:48:41 -04:00
parent 53de25120a
commit 8fc88ffc14
117 changed files with 4717 additions and 1443 deletions
@@ -0,0 +1,56 @@
+# Runtime and ops surface (post Phases 1–2)
+
+Design note for operators and engineers. Reflects the **current** intended production topology.
+
+## Processes
+
+| Process | Role | Notes |
+|---------|------|--------|
+| `api` | HTTP API | Do **not** set `ENABLE_EMBEDDED_JOBS=true` when running multiple API replicas |
+| `api-worker` | Outbox + scheduled jobs | Uses DB leases + Redis leader lock for cron |
+| Frontends | homepage, carplace, dashboard, admin | Unchanged modular apps |
+| postgres | System of record | |
+| redis | Rate limit, idempotency, pub/sub, locks | Required for replica-safe behavior |
+| object storage | Optional S3/MinIO | Prefer over local disk for multi-replica files |
+
+## HTTP ops endpoints
+
+| Method | Path | Purpose |
+|--------|------|---------|
+| GET | `/health` | Liveness |
+| GET | `/ready` | Readiness: database, redis, storage |
+| GET | `/metrics` | Prometheus-style counters/gauges (latency, status, outbox pending/published) |
+| GET | `/api/v1/openapi.json` | OpenAPI document |
+| GET | `/docs` | Swagger UI (when enabled) |
+
+Worker (optional):
+
+| Env | Behavior |
+|-----|----------|
+| `WORKER_METRICS_PORT=<port>` | Worker listens for `GET /metrics` and `GET /health` |
+| `0` / unset | No worker HTTP listener |
+
+## Important environment knobs
+
+Documented in `.env.example`:
+
+- `ENABLE_EMBEDDED_JOBS` — default false in production examples  
+- `RATE_LIMIT_STORE` / `IDEMPOTENCY_STORE` — prefer `redis`  
+- `FILE_STORAGE_DRIVER` — `local` or `s3` (+ `S3_*`)  
+- `TRUSTED_FORWARD_HEADERS` — default false; see `docs/ops/proxy-trust.md`  
+- `ADMIN_FRESH_2FA_MAX_AGE_MS` — default `1800000` (30 minutes)  
+- `WORKER_METRICS_PORT` — optional  
+
+## Observability design
+
+- Access logs: structured JSON via morgan in `app.ts`  
+- In-process series: `apps/api/src/lib/opsMetrics.ts`  
+- Durable outbox gauges: API `/metrics` also counts PENDING / PUBLISHED rows from PostgreSQL so scrapes remain useful when jobs run only on the worker  
+
+## Failure / release runbooks
+
+- Chaos helper: `scripts/chaos/failure-injection.sh`  
+- Soak: `npm run test:soak` → `scripts/load/soak-probe.mjs`  
+- Canary / rollback: `docs/ops/canary-rollback.md`  
+- Key rotation: `docs/ops/key-rotation-drill.md`  
+- Backup smoke: `scripts/backup-restore-smoke-check.sh`