bluewave-labs / bluewave-labs/Checkmate

Monitors randomly stop working - monitors locked but never unlocked?

Open
#3,073 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
10.8k
Forks
1.2k
Avg merge
1d 4h
Merged PRs (30d)
68

Description

**Describe the bug**
Monitors randomly break causing them to not alert anymore on incidents. I also cannot get them back to work without removing them and re-adding them. Then they work for a while again, but end up breaking again after a while.

I have read some other (closed) issues suggesting persistence issues, but this is not the case here:
- I only have Checkmate running on 1 dedicated machine
- The database is a bind-mount to the file system, so not a volume:
``` - /docker_compose/checkmate/mongo/data:/data/db ```
- The server has not been down or turned off

**To Reproduce**
I cannot reproduce it at will.

**Expected behavior**
I expect the monitors to keep running and be fault-tolerant

**Screenshots**
Image

---

Image

---

Image

---

Image

---

Image

---

Image

**Additional context**
From the Job queue, it seems like the entries for these monitors are locked, but never unlocked. See screenshot.

Contributor guide

Open the contributing guide

Research direction

The issue provides no file or test entry point; begin by tracing the Job queue entries for affected monitors and the monitor lock/unlock lifecycle. Reproduce or inspect the locked state if possible, and consider the issue done only when monitors continue alerting without removal and re-addition and queue locks are released.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.