bluewave-labs / bluewave-labs/Checkmate
Monitors randomly stop working - monitors locked but never unlocked?
- Dominant language
- TypeScript
- Stars
- 10.8k
- Forks
- 1.2k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 68
Description
**Describe the bug**
Monitors randomly break causing them to not alert anymore on incidents. I also cannot get them back to work without removing them and re-adding them. Then they work for a while again, but end up breaking again after a while.
I have read some other (closed) issues suggesting persistence issues, but this is not the case here:
- I only have Checkmate running on 1 dedicated machine
- The database is a bind-mount to the file system, so not a volume:
``` - /docker_compose/checkmate/mongo/data:/data/db ```
- The server has not been down or turned off
**To Reproduce**
I cannot reproduce it at will.
**Expected behavior**
I expect the monitors to keep running and be fault-tolerant
**Screenshots**
---
---
---
---
---
**Additional context**
From the Job queue, it seems like the entries for these monitors are locked, but never unlocked. See screenshot.
Contributor guide
Research direction
The issue provides no file or test entry point; begin by tracing the Job queue entries for affected monitors and the monitor lock/unlock lifecycle. Reproduce or inspect the locked state if possible, and consider the issue done only when monitors continue alerting without removal and re-addition and queue locks are released.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100