Production Incident Remediation — Ops Follow-ups

This document covers remediation items that cannot be completed from application code alone. Deploy the heartbeat / filter / rate-limit code changes first, then work through this list.

Deployment wave 1 (application)

Back up production database and current release.
Deploy:
- throttled /session/ping-activity client (public/assets/js/session_timeout.js)
- filter exclusions for timezone + school-year writable on session endpoints
- lightweight SessionTimeoutController
- route rate limit apiratelimit:5,60 on ping endpoints
Monitor for 30–60 minutes:
- ping-activity requests/minute
- HTTP 429 / 500 / 503 rates
- MySQL connection errors

3. MySQL `Operation not permitted`

Do not assume a bad password. Check Hostinger / server logs for the incident window 04:12:03–06:21:08 and capture:

SHOW VARIABLES LIKE 'max_connections';
SHOW STATUS LIKE 'Threads_connected';
SHOW STATUS LIKE 'Max_used_connections';
SHOW STATUS LIKE 'Aborted_connects';

Also inspect:

PHP-FPM / LiteSpeed worker limits
Hostinger resource throttling / inodes / CPU
firewall / socket restrictions
temporary MySQL outages

Document the confirmed root cause here once known:

Cause:
Evidence:
Fix applied:

5. Missing tables / migrations

After backup:

php spark migrate:status
php spark migrate

Confirm:

settings exists
user_preferences exists
no new table doesn't exist errors

Migration 2026-08-20-010000_CreateSettingsAndUserPreferencesTables creates both tables if missing.

7. SMTP authentication

Credentials must live in environment configuration (mail.SMTP* / SMTP_*), not in app/Config/Email.php.

Verify production env:

SMTP host / port / encryption
username + app password
from address / provider restrictions

If auth recently failed, rotate the Gmail app password and update production env only. Send one controlled registration or student-removal email from production and confirm success.

10. Monitoring / alerts

Track at least:

HTTP RPS
/session/ping-activity requests/minute
HTTP 500 and 429 rates
DB connection failures
active MySQL connections
PHP worker usage
SMTP failures

Suggested initial alerts:

DB connection errors > 5/minute
500 responses > 1% of requests
ping-activity above expected session-based threshold
MySQL connections > 80% of max_connections

Verification checklist

No overlapping ping requests from one tab
Failed pings use exponential backoff
Heartbeat skips timezone / school-year / settings queries
Server-side ping rate limiting returns 429 when exceeded
MySQL Operation not permitted cause identified
settings + user_preferences exist; migrations current
SMTP auth succeeds
Zero-value invoices return controlled 422 (no null setJSON)
Monitoring / alerts enabled

2.9 KiB Raw Permalink Blame History Unescape Escape