bluewave-labs / bluewave-labs/Checkmate

Reduce MongoDB healthcheck overhead in Docker Compose

Open
#3,877 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
10.8k
Forks
1.2k
Avg merge
1d 4h
Merged PRs (30d)
68

Description

**Describe the bug**

The MongoDB health check in both Docker Compose configurations starts a new `mongosh` process every five seconds. Because the health-check process runs inside the MongoDB container, its startup cost is included in the container's CPU usage and produces recurring CPU spikes even when database activity is low.

**To Reproduce**

1. Start Checkmate with Docker Compose.
2. Run a small set of active monitors.
3. Observe the MongoDB container with `docker stats`.
4. Compare the periodic container CPU spikes with MongoDB operation counters or `mongotop` output.

In a local test with six active monitors, the MongoDB container periodically reached 56.64% and 47.10% CPU. During a 30-second sample, MongoDB averaged 0.43 inserts/s, 6.87 queries/s, 2.33 updates/s, and 4.43 commands/s, while collection activity measured by `mongotop` remained within a few milliseconds per second.

**Expected behavior**

The health check should retain the semantic `mongosh` ping while avoiding unnecessary steady-state process startups. Startup checks should remain frequent so `depends_on: condition: service_healthy` is not delayed.

**Desktop (please complete the following information):**

- Environment: Docker Desktop on macOS
- MongoDB image: `mongo:8.0`
- Checkmate branch: `develop`

**Additional context**

A 30-second steady-state interval with a 30-second start period and a two-second start interval preserves fast startup detection while reducing steady-state `mongosh` executions by approximately 83%. A five-second timeout and five retries retain a reasonable failure-detection window.

**Self-hosted reproduction**

The same behavior was reproduced on a Dokploy-managed self-hosted Checkmate deployment running on a 2-vCPU, 2-GB RAM server. The MongoDB container was limited to 0.75 CPU and 512 MB RAM.

With the current five-second healthcheck interval, a 20-sample observation recorded:

- 4.56% median MongoDB container CPU
- 77.85% p95 MongoDB container CPU
- 78.44% maximum MongoDB container CPU

The application container remained comparatively quiet during the same observation. This supports the local finding that the recurring spikes are dominated by healthcheck process startup rather than monitor write volume.

Recurring MongoDB CPU spikes with the five-second healthcheck

A proposed fix and before/after validation are available in #3878.

Contributor guide

Open the contributing guide

Research direction

The issue concerns the MongoDB healthcheck in both Docker Compose configurations; start by locating those healthcheck definitions and compare them with PR #3878. Verify that the proposed timing preserves fast startup detection while reducing steady-state mongosh executions and recurring container CPU spikes.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker-compose, mongodb
Domain
devops, infrastructure
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.