# Runtime and ops surface (post Phases 1–2)

Design note for operators and engineers. Reflects the **current** intended production topology.

## Processes

| Process | Role | Notes |
|---------|------|--------|
| `api` | HTTP API | Do **not** set `ENABLE_EMBEDDED_JOBS=true` when running multiple API replicas |
| `api-worker` | Outbox + scheduled jobs | Uses DB leases + Redis leader lock for cron |
| Frontends | homepage, carplace, dashboard, admin | Unchanged modular apps |
| postgres | System of record | |
| redis | Rate limit, idempotency, pub/sub, locks | Required for replica-safe behavior |
| object storage | Optional S3/MinIO | Prefer over local disk for multi-replica files |

## HTTP ops endpoints

| Method | Path | Purpose |
|--------|------|---------|
| GET | `/health` | Liveness |
| GET | `/ready` | Readiness: database, redis, storage |
| GET | `/metrics` | Prometheus-style counters/gauges (latency, status, outbox pending/published) |
| GET | `/api/v1/openapi.json` | OpenAPI document |
| GET | `/docs` | Swagger UI (when enabled) |

Worker (optional):

| Env | Behavior |
|-----|----------|
| `WORKER_METRICS_PORT=<port>` | Worker listens for `GET /metrics` and `GET /health` |
| `0` / unset | No worker HTTP listener |

## Important environment knobs

Documented in `.env.example`:

- `ENABLE_EMBEDDED_JOBS` — default false in production examples  
- `RATE_LIMIT_STORE` / `IDEMPOTENCY_STORE` — prefer `redis`  
- `FILE_STORAGE_DRIVER` — `local` or `s3` (+ `S3_*`)  
- `TRUSTED_FORWARD_HEADERS` — default false; see `docs/ops/proxy-trust.md`  
- `ADMIN_FRESH_2FA_MAX_AGE_MS` — default `1800000` (30 minutes)  
- `WORKER_METRICS_PORT` — optional  

## Observability design

- Access logs: structured JSON via morgan in `app.ts`  
- In-process series: `apps/api/src/lib/opsMetrics.ts`  
- Durable outbox gauges: API `/metrics` also counts PENDING / PUBLISHED rows from PostgreSQL so scrapes remain useful when jobs run only on the worker  

## Failure / release runbooks

- Chaos helper: `scripts/chaos/failure-injection.sh`  
- Soak: `npm run test:soak` → `scripts/load/soak-probe.mjs`  
- Canary / rollback: `docs/ops/canary-rollback.md`  
- Key rotation: `docs/ops/key-rotation-drill.md`  
- Backup smoke: `scripts/backup-restore-smoke-check.sh`