bluewave-labs / bluewave-labs/Checkmate
Reduce MongoDB healthcheck overhead in Docker Compose
- Dominant language
- TypeScript
- Stars
- 10.8k
- Forks
- 1.2k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 68
Description
**Describe the bug**
The MongoDB health check in both Docker Compose configurations starts a new `mongosh` process every five seconds. Because the health-check process runs inside the MongoDB container, its startup cost is included in the container's CPU usage and produces recurring CPU spikes even when database activity is low.
**To Reproduce**
1. Start Checkmate with Docker Compose.
2. Run a small set of active monitors.
3. Observe the MongoDB container with `docker stats`.
4. Compare the periodic container CPU spikes with MongoDB operation counters or `mongotop` output.
In a local test with six active monitors, the MongoDB container periodically reached 56.64% and 47.10% CPU. During a 30-second sample, MongoDB averaged 0.43 inserts/s, 6.87 queries/s, 2.33 updates/s, and 4.43 commands/s, while collection activity measured by `mongotop` remained within a few milliseconds per second.
**Expected behavior**
The health check should retain the semantic `mongosh` ping while avoiding unnecessary steady-state process startups. Startup checks should remain frequent so `depends_on: condition: service_healthy` is not delayed.
**Desktop (please complete the following information):**
- Environment: Docker Desktop on macOS
- MongoDB image: `mongo:8.0`
- Checkmate branch: `develop`
**Additional context**
A 30-second steady-state interval with a 30-second start period and a two-second start interval preserves fast startup detection while reducing steady-state `mongosh` executions by approximately 83%. A five-second timeout and five retries retain a reasonable failure-detection window.
**Self-hosted reproduction**
The same behavior was reproduced on a Dokploy-managed self-hosted Checkmate deployment running on a 2-vCPU, 2-GB RAM server. The MongoDB container was limited to 0.75 CPU and 512 MB RAM.
With the current five-second healthcheck interval, a 20-sample observation recorded:
- 4.56% median MongoDB container CPU
- 77.85% p95 MongoDB container CPU
- 78.44% maximum MongoDB container CPU
The application container remained comparatively quiet during the same observation. This supports the local finding that the recurring spikes are dominated by healthcheck process startup rather than monitor write volume.
A proposed fix and before/after validation are available in #3878.
Contributor guide
Research direction
The issue concerns the MongoDB healthcheck in both Docker Compose configurations; start by locating those healthcheck definitions and compare them with PR #3878. Verify that the proposed timing preserves fast startup detection while reducing steady-state mongosh executions and recurring container CPU spikes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker-compose, mongodb
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100