# Production Incident Remediation — Ops Follow-ups

This document covers remediation items that cannot be completed from application code alone.
Deploy the heartbeat / filter / rate-limit code changes first, then work through this list.

## Deployment wave 1 (application)

1. Back up production database and current release.
2. Deploy:
   - throttled `/session/ping-activity` client (`public/assets/js/session_timeout.js`)
   - filter exclusions for timezone + school-year writable on session endpoints
   - lightweight `SessionTimeoutController`
   - route rate limit `apiratelimit:5,60` on ping endpoints
3. Monitor for 30–60 minutes:
   - ping-activity requests/minute
   - HTTP 429 / 500 / 503 rates
   - MySQL connection errors

## 3. MySQL `Operation not permitted`

Do **not** assume a bad password. Check Hostinger / server logs for the incident window
`04:12:03`–`06:21:08` and capture:

```sql
SHOW VARIABLES LIKE 'max_connections';
SHOW STATUS LIKE 'Threads_connected';
SHOW STATUS LIKE 'Max_used_connections';
SHOW STATUS LIKE 'Aborted_connects';
```

Also inspect:

- PHP-FPM / LiteSpeed worker limits
- Hostinger resource throttling / inodes / CPU
- firewall / socket restrictions
- temporary MySQL outages

Document the confirmed root cause here once known:

- Cause:
- Evidence:
- Fix applied:

## 5. Missing tables / migrations

After backup:

```bash
php spark migrate:status
php spark migrate
```

Confirm:

- `settings` exists
- `user_preferences` exists
- no new `table doesn't exist` errors

Migration `2026-08-20-010000_CreateSettingsAndUserPreferencesTables` creates both tables if missing.

## 7. SMTP authentication

Credentials must live in environment configuration (`mail.SMTP*` / `SMTP_*`), not in
`app/Config/Email.php`.

Verify production env:

- SMTP host / port / encryption
- username + app password
- from address / provider restrictions

If auth recently failed, rotate the Gmail app password and update production env only.
Send one controlled registration or student-removal email from production and confirm success.

## 10. Monitoring / alerts

Track at least:

- HTTP RPS
- `/session/ping-activity` requests/minute
- HTTP 500 and 429 rates
- DB connection failures
- active MySQL connections
- PHP worker usage
- SMTP failures

Suggested initial alerts:

- DB connection errors > 5/minute
- 500 responses > 1% of requests
- ping-activity above expected session-based threshold
- MySQL connections > 80% of `max_connections`

## Verification checklist

- [ ] No overlapping ping requests from one tab
- [ ] Failed pings use exponential backoff
- [ ] Heartbeat skips timezone / school-year / settings queries
- [ ] Server-side ping rate limiting returns 429 when exceeded
- [ ] MySQL `Operation not permitted` cause identified
- [ ] `settings` + `user_preferences` exist; migrations current
- [ ] SMTP auth succeeds
- [ ] Zero-value invoices return controlled 422 (no null `setJSON`)
- [ ] Monitoring / alerts enabled