ColoredCow / ColoredCow/monitor

monitor:aggregate-check-metrics runs out of memory (exit 137) in production — OOM takes down MySQL and hangs the server

Open
#83 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
PHP
Stars
5
Forks
1
PR merge metrics
No merged PRs in 30d

Description

## Summary
The scheduled command `monitor:aggregate-check-metrics` is exhausting memory on the production host (monitor.coloredcow.com). It gets OOM-killed (exit code **137**), and because the box is memory-constrained, the OOM-killer then takes down **MySQL**, which cascades into failed uptime checks and a fully unresponsive server. This recurred repeatedly on 2026-07-03 (site down, SSH/HTTP timing out).

## Impact
- Production monitoring app (`monitor.coloredcow.com`) goes fully down and needs a manual reboot / stop-start to recover.
- When it's down, **we lose uptime monitoring for the whole fleet** — the tool that's supposed to catch outages is itself offline.
- Recurs on every scheduler cycle until the process is stopped.

## Evidence (production `laravel.log`, 2026-07-03)
```
[08:18:19] production.ERROR: Scheduled command [... monitor:aggregate-check-metrics] failed with exit code [137].
[10:55:03] production.ERROR: SQLSTATE[HY000] [2002] Connection refused (Connection: mysql, Host: 127.0.0.1 ...)
[10:55:03] production.ERROR: Scheduled command [... monitor:check-uptime] failed with exit code [1].
```
Exit code **137 = 128 + 9 (SIGKILL)** — the kernel OOM-killer terminated the process for using too much memory. The `Connection refused` immediately after is MySQL going down under the same memory pressure.

## Root cause
`monitor:aggregate-check-metrics` appears to load a large result set into memory (aggregating check/metric history in PHP) rather than aggregating in the database or streaming. On a small host running MySQL + PHP-FPM alongside it, that pushes total memory usage past the limit → OOM.

Likely introduced/exposed by #82 (organization-dashboard) — the metrics aggregation added for the new dashboard.

## Proposed fix (app side)
1. **Aggregate in SQL, not PHP** — replace any `->get()`/`->all()`/large-collection aggregation with `selectRaw(...)` + `GROUP BY`, or process in batches via `chunkById()` / a lazy cursor so the whole dataset is never held in memory at once.
2. **Bound memory per run** — cap the time window / rows processed per invocation (e.g. aggregate incrementally rather than reprocessing all history each run).
3. **Lower `concurrent_checks`** in `config/uptime-monitor.php` from `10` (reduces peak memory during `monitor:check-uptime` too).
4. **Make the scheduler resilient** — ensure `withoutOverlapping()` on these commands so runs can't stack, and that one command failing doesn't abort the rest of the schedule.
5. (Optional) Move heavy work off the inline `sync` queue (`QUEUE_CONNECTION=sync`) if a worker can be run within the host's memory budget.

## Acceptance criteria
- [ ] `monitor:aggregate-check-metrics` completes within a bounded memory ceiling (e.g. < 128 MB) regardless of history size.
- [ ] No `exit code [137]` / OOM entries in `laravel.log` after deploy.
- [ ] Peak memory during a full scheduler cycle (check-uptime + aggregate) verified to stay within the host's available RAM.

## Environment
- Production monitor host: small single instance (~1 GB RAM), MySQL + PHP-FPM 8.3 co-located.
- Package: `spatie/laravel-uptime-monitor`, 25 active monitors.
- **Infra mitigation already applied** (2026-07-03): root volume grown and a **2 GB swap file added**, so the box now degrades under memory pressure instead of hard-hanging. This makes the command *survivable* but not efficient — the app-side fix above is still needed so aggregation isn't chronically swapping.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.