EbookFoundation / EbookFoundation/regluit-provisioning

Add StatusCheckFailed CloudWatch alarms on test + dj42 EC2 (silent-outage gap, 5/21 OOM cascade)

Open
#36 0 comments 0 reactions 0 assignees View on GitHub
theme:performance
Dominant language
Shell
Stars
6
Forks
8
Avg merge
1h 13m
Merged PRs (30d)
2

Description

**Why**: On 2026-05-21, `test.unglue.it` went into an OOM cascade (Apache killed → guest network stack hung) and sat impaired for **5 days** before discovery, because:

- **No CloudWatch alarms** are configured on the EC2 instance (`StatusCheckFailed`, CPU, memory — none)
- `disable_admin_emails: true` on test (and dj42) per Ansible config — error mail to Eric is suppressed by design
- SolarWinds monitors prod only; test/dj42 aren't watched

Diagnostic trail from the 2026-05-26 healthcheck:

| Layer | State |
|---|---|
| `SystemStatus` (hypervisor) | ok |
| `InstanceStatus` (guest reachability) | **failed since 2026-05-21T15:51Z** |
| CPU 5/21 16:45 → 18:35 | sustained ~58–60% (runaway) |
| CPU 5/21 18:40 → present | flatline 0.36% (guest hung) |
| Console screenshot | `Out of memory: Killed process 13064 (apache2) UID:33` |

Console screenshot saved locally (`/tmp/test-unglue-console.jpg`). Can attach if useful.

**Minimum fix (this issue)**

Add a CloudWatch alarm to each non-prod EC2 in AWS account `439256357102` (Gluejar member, region `us-east-1`):

| Instance | Tag | Instance type |
|---|---|---|
| `i-0bd911f9ae13310b6` | `test-unglue-it` | t3.medium |
| `i-035d248ac4b18ef60` | `dj42-unglue-it` | t3.medium |

Alarm spec:
- Metric: `AWS/EC2 StatusCheckFailed_Instance`
- Statistic: Maximum, period 60s
- Threshold: `> 0` for 15 minutes (15 consecutive periods, or 3× 5-min)
- Treat missing data: `breaching`
- Action: SNS topic → raymond.yee@gmail.com (and optionally Eric)

**Out of scope (separate work)**

- Investigate *why* test OOMs (Apache config? Celery? memory leak? t3.medium undersized?). Different problem; alarm is the alarm.
- Disk + memory alarms (CloudWatch agent install required for memory; can be a follow-up)
- Whether `disable_admin_emails: true` should be reconsidered on test/dj42 (tracked separately — see handoff queue)
- Prod (`i-038105b781b514601` / `prod7_py3`) already has SolarWinds coverage; in scope only if we want belt-and-suspenders

**Implementation note**

Probably a single `boto3`/`awscli` script or a small task in this playbook (`monitoring/` role?). Open to whichever pattern fits the existing provisioning conventions — TBD with reviewer.

**Provenance**

Issue body drafted by Claude Code from a live healthcheck session; reviewed by Raymond before filing.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.