Monitoring: Adjust health-check thresholds

Context

Prod emits recurring "downtime detected" alerts. Alert noise traced to threshold config: staging (backend, guardrails, frontend) uses threshold 3 and produces no false alerts; prod is on a different value. Request timeout is 15s, exponential backoff ~3–4s then ~8s, health check every 1 min.

Decisions

• Set prod health-check threshold to 3. Caveat: if the health endpoint genuinely isn't responding (prod is intermittently loaded), the threshold change only silences the symptom — write the real hypothesis (FastAPI thread saturation at ~40 threads / ~50–100 concurrent requests) into the issue so a green dashboard isn't misread as solved.
• When a downtime alarm fires, first hit ~5–6 known endpoints; if they return 200, treat as a false alarm and suppress.

Verification (before assuming bot-flood root cause)

• For each downtime window, check logs + Sentry: was the server actually down, did real LLM calls come through and with what status, did bot call volume spike? Confirm against Prajna's root cause.
• Pull logs for the next alert window and confirm actual server behavior.

Related

• A Sentry cron monitor is also firing on staging while staging is green — determine whether Sentry is misbehaving, it's the dev-env uptime check, or the notification setup is wrong.