ColoredCow / ColoredCow/monitor
monitor:aggregate-check-metrics runs out of memory (exit 137) in production — OOM takes down MySQL and hangs the server
- Dominant language
- PHP
- Stars
- 5
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
The scheduled command `monitor:aggregate-check-metrics` is exhausting memory on the production host (monitor.coloredcow.com). It gets OOM-killed (exit code **137**), and because the box is memory-constrained, the OOM-killer then takes down **MySQL**, which cascades into failed uptime checks and a fully unresponsive server. This recurred repeatedly on 2026-07-03 (site down, SSH/HTTP timing out).
## Impact
- Production monitoring app (`monitor.coloredcow.com`) goes fully down and needs a manual reboot / stop-start to recover.
- When it's down, **we lose uptime monitoring for the whole fleet** — the tool that's supposed to catch outages is itself offline.
- Recurs on every scheduler cycle until the process is stopped.
## Evidence (production `laravel.log`, 2026-07-03)
```
[08:18:19] production.ERROR: Scheduled command [... monitor:aggregate-check-metrics] failed with exit code [137].
[10:55:03] production.ERROR: SQLSTATE[HY000] [2002] Connection refused (Connection: mysql, Host: 127.0.0.1 ...)
[10:55:03] production.ERROR: Scheduled command [... monitor:check-uptime] failed with exit code [1].
```
Exit code **137 = 128 + 9 (SIGKILL)** — the kernel OOM-killer terminated the process for using too much memory. The `Connection refused` immediately after is MySQL going down under the same memory pressure.
## Root cause
`monitor:aggregate-check-metrics` appears to load a large result set into memory (aggregating check/metric history in PHP) rather than aggregating in the database or streaming. On a small host running MySQL + PHP-FPM alongside it, that pushes total memory usage past the limit → OOM.
Likely introduced/exposed by #82 (organization-dashboard) — the metrics aggregation added for the new dashboard.
## Proposed fix (app side)
1. **Aggregate in SQL, not PHP** — replace any `->get()`/`->all()`/large-collection aggregation with `selectRaw(...)` + `GROUP BY`, or process in batches via `chunkById()` / a lazy cursor so the whole dataset is never held in memory at once.
2. **Bound memory per run** — cap the time window / rows processed per invocation (e.g. aggregate incrementally rather than reprocessing all history each run).
3. **Lower `concurrent_checks`** in `config/uptime-monitor.php` from `10` (reduces peak memory during `monitor:check-uptime` too).
4. **Make the scheduler resilient** — ensure `withoutOverlapping()` on these commands so runs can't stack, and that one command failing doesn't abort the rest of the schedule.
5. (Optional) Move heavy work off the inline `sync` queue (`QUEUE_CONNECTION=sync`) if a worker can be run within the host's memory budget.
## Acceptance criteria
- [ ] `monitor:aggregate-check-metrics` completes within a bounded memory ceiling (e.g. < 128 MB) regardless of history size.
- [ ] No `exit code [137]` / OOM entries in `laravel.log` after deploy.
- [ ] Peak memory during a full scheduler cycle (check-uptime + aggregate) verified to stay within the host's available RAM.
## Environment
- Production monitor host: small single instance (~1 GB RAM), MySQL + PHP-FPM 8.3 co-located.
- Package: `spatie/laravel-uptime-monitor`, 25 active monitors.
- **Infra mitigation already applied** (2026-07-03): root volume grown and a **2 GB swap file added**, so the box now degrades under memory pressure instead of hard-hanging. This makes the command *survivable* but not efficient — the app-side fix above is still needed so aggregation isn't chronically swapping.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.