[Bug]: Worker stops picking up tasks for duplicate key
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 69
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
Monitor had timeout and has not reported in multiple days
**Database**
- postgresql
**To Reproduce**
Not sure how to reproduce looks like there was some sort of error in Redis/Postgres and now a duplicate key exists and the worker is no longer picking up from the queue to check the site
**Expected behavior**
The system should be able to recover even with duplicate key and continue health checks
**Screenshots**
^ Can see that the worker stopped 3 days ago after reporting DOWN
Producer logs showing duplicate entry in the Queue service
Ingester showing both duplicate in queue service and a pg error
Worker successfully picking up one monitor but not the one that is erroring out
**Additional context**
This is running inside Kubernetes using a Postgresql and Redis from Bitnami but I would imagine this scenario is not related and seems to be more of an issue with inserting / retrieving information from Redis/Postgresql
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue involves a worker queue system using Redis and PostgreSQL. Examine the queue service and worker code for handling duplicate keys and database errors. Look at the ingester and producer logs to understand the failure flow. The fix likely requires modifying the error handling and recovery logic in the queue processing components.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes, postgresql, redis
- Domain
- backend, databases, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100