EbookFoundation / EbookFoundation/regluit-provisioning

Monitoring: SES reputation + enforcement state

Open
#28 1 comment 0 reactions 0 assignees View on GitHub
theme:infra
Dominant language
Shell
Stars
6
Forks
8
Avg merge
1h 13m
Merged PRs (30d)
2

Description

## Background

The 2026-04-23 email flood (see [security-private#11](https://github.com/EbookFoundation/security-private/issues/11)) sent **~375K emails over 4 days with ~76–87% daily bounce rates** — well above AWS SES's 5% "under review" and 10% "shutdown" thresholds. The account escaped enforcement only because SES reputation calculations lag ~72 hours and the flood ended before the next review cycle.

**Existing monitoring did not detect this.** Eric noticed via his inbox. No CloudWatch alarm exists for any SES metric. This issue documents the systematization of SES monitoring.

## What's missing

Searched existing CloudWatch alarms in account `439256357102` (us-east-1) on 2026-04-23:

| Namespace | Alarms | Covers SES? |
|---|---|---|
| AWS/EC2 | 7 | No |
| AWS/RDS | 2 | No |
| AWS/SES | **0** | **No** |

Many of the existing EC2 alarms point to instance IDs that no longer exist (INSUFFICIENT_DATA state) — separate cleanup issue.

## Proposal

### Tier 1 — immediate (covered by companion PRs)

- CloudWatch alarm: \`Reputation.BounceRate > 5%\` → SNS → email (warning)
- CloudWatch alarm: \`Reputation.BounceRate > 10%\` → SNS → email (critical)
- CloudWatch alarm: \`Reputation.ComplaintRate > 0.1%\` → SNS → email (complaint-rate threshold)

These alarms will be created manually via AWS CLI once an SES alerts SNS topic exists.

### Tier 2 — infrastructure-as-code

Move the alarms into Ansible/Terraform so they're versioned, reproducible, and part of the standard deploy. Candidate: add a \`monitoring\` role in \`regluit-provisioning\` that declares CloudWatch alarms. This also fits cleanup of the stale EC2 alarms from defunct instances.

### Tier 3 — non-metric state checks

\`EnforcementStatus\` from \`aws sesv2 get-account\` is **not a CloudWatch metric** — it can only be polled. Options:

1. **Scheduled Lambda** (every ~4 hours): calls \`sesv2 get-account\`, publishes a custom CloudWatch metric (\`CustomSES/EnforcementStatus\` = 0/1/2 for HEALTHY/REVIEW/SHUTDOWN), and triggers alarm if non-zero.
2. **Cron on test or prod host**: same logic in bash, writes to CloudWatch via \`aws cloudwatch put-metric-data\`.
3. **Nothing** — rely on SES sending the degraded-reputation email directly to the account owner.

(3) is cheapest; (1) is most robust. Probably (3) is enough given this is a once-per-decade event, but capturing it in a runbook is important.

### Tier 4 — runbook

Document in \`regluit-provisioning/docs/\` (or wherever deploy docs live):

- How to check SES account state: \`aws sesv2 get-account\` one-liner
- How to check reputation metrics: \`aws cloudwatch get-metric-statistics --namespace AWS/SES --metric-name Reputation.BounceRate ...\`
- What to do if \`EnforcementStatus\` changes: immediate triage steps, who to escalate to
- What to do if account is paused (SendingEnabled=false): recovery procedure, AWS case opening template
- How to spot a flood early: \`Bounce\` metric daily count, with guidance on what "normal" looks like

## Reference commands (to include in runbook)

\`\`\`bash
# Daily health check
AWS_PROFILE=ebookfoundation aws sesv2 get-account --region us-east-1 \\
--query '{Status:EnforcementStatus,Enabled:SendingEnabled,Quota24h:SendQuota.Max24HourSend,RatePerSec:SendQuota.MaxSendRate,Sent24h:SendQuota.SentLast24Hours}'

# Send stats (15-day rolling window, no setup required)
AWS_PROFILE=ebookfoundation aws ses get-send-statistics --region us-east-1

# Bounce rate trend
AWS_PROFILE=ebookfoundation aws cloudwatch get-metric-statistics \\
--namespace AWS/SES --metric-name Reputation.BounceRate \\
--start-time \$(date -u -v-7d +%Y-%m-%dT%H:%M:%SZ) \\
--end-time \$(date -u +%Y-%m-%dT%H:%M:%SZ) \\
--period 86400 --statistics Maximum Average --region us-east-1
\`\`\`

## AWS thresholds (for reference)

| Metric | Warning threshold | Shutdown threshold |
|---|---|---|
| Bounce rate | 5% (account placed under review) | 10% (sending paused) |
| Complaint rate | 0.1% (warning) | 0.5% (sending paused) |

Both are rolling 15-day averages. Brief spikes are tolerated; sustained elevated rates trigger action.

## Related

- Incident that surfaced this gap: [security-private#11](https://github.com/EbookFoundation/security-private/issues/11)
- Remediation PRs: [Gluejar/regluit#1128](https://github.com/Gluejar/regluit/pull/1128), [regluit-provisioning#27](https://github.com/EbookFoundation/regluit-provisioning/pull/27)
- Stale EC2 alarms (separate cleanup): not yet filed

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.