Monitors stop checking forever after a task gets archived
- Lenguaje dominante
- Go
- Estrellas
- 1.2k
- Forks
- 69
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
Ran into this on the postgres bundle image. Some monitors just stopped generating heartbeats even though they showed as active. Checked the DB, no rows. Checked the logs and every 5 min I'd see:
```
msg="Task already queued (duplicate prevented by unique constraint)"
task_type=monitor:healthcheck
unique_key=healthcheck:
```
Digging into Redis, the task was sitting in `asynq:{healthcheck}:archived` since the last time it hit `context deadline exceeded` and asynq gave up. Since the producer uses the monitor uuid as the asynq TaskID, and asynq's TaskID uniqueness blocks enqueue if a task with that ID exists in any state (archived included, and archived has no TTL), the monitor is basically locked out forever.
What made it worse in our case: I had edited the monitor (changed the keyword, URL) after the task got archived, so the archived payload still had the old broken config, and the new config never got a chance to run.
I checked across our install and found 36 monitors in this exact state, oldest one stuck for ~31h. Zero alerts because the last-known status just persists.
Workaround that fixed it:
```
docker exec sh -c 'TIDS=$(redis-cli ZRANGE "asynq:{healthcheck}:archived" 0 -1); for T in $TIDS; do redis-cli DEL "asynq:{healthcheck}:t:$T"; redis-cli ZREM "asynq:{healthcheck}:archived" "$T"; done'
docker restart
```
After that producer re-enqueued and heartbeats came back within the interval.
I ended up putting a cron on the host that runs that snippet every 5 min, but feels like it should be handled upstream. Options that come to mind:
- purge the task across all asynq states when a monitor is updated/deleted
- use `asynq.Unique(ttl)` instead of TaskID as the idempotency key
- reaper loop in the producer that drops archived entries older than X
Log reference for the dupe check is `infra/queue.go:276`.
Not urgent for me anymore since the cron catches it, but wanted to flag it because it's silent and easy to miss.
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Línea de trabajo
The bug is in the task queue logic where archived tasks block new ones. Start by examining `infra/queue.go` around line 276 to understand the duplicate prevention. Look at how tasks are enqueued with the monitor UUID as the TaskID and how they end up in the archived set in Redis. A fix must ensure tasks are purged from all states on monitor updates or use a different idempotency strategy. Testing will involve simulating the archived state and verifying new tasks can be enqueued.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- go, redis
- Área
- backend, devops, observability-sre
- Tipo de issue
- Error
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Estado de actividad
- Estancado
- Claridad
- Bien especificado
- Aptitud para principiantes
- 40/100