0xfurai / 0xfurai/peekaping

[Bug]: Worker stops picking up tasks for duplicate key

Open
#237 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
1.2k
Forks
69
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
Monitor had timeout and has not reported in multiple days

**Database**
- postgresql

**To Reproduce**
Not sure how to reproduce looks like there was some sort of error in Redis/Postgres and now a duplicate key exists and the worker is no longer picking up from the queue to check the site

**Expected behavior**
The system should be able to recover even with duplicate key and continue health checks

**Screenshots**

Image

^ Can see that the worker stopped 3 days ago after reporting DOWN

Image

Producer logs showing duplicate entry in the Queue service

Image

Ingester showing both duplicate in queue service and a pg error

Image

Worker successfully picking up one monitor but not the one that is erroring out

**Additional context**
This is running inside Kubernetes using a Postgresql and Redis from Bitnami but I would imagine this scenario is not related and seems to be more of an issue with inserting / retrieving information from Redis/Postgresql

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue involves a worker queue system using Redis and PostgreSQL. Examine the queue service and worker code for handling duplicate keys and database errors. Look at the ingester and producer logs to understand the failure flow. The fix likely requires modifying the error handling and recovery logic in the queue processing components.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, postgresql, redis
Domain
backend, databases, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.