codingjoe / codingjoe/threadmill
Silently dropped task ids when the task hash is missing in acquire.lua and mover.lua
- Lenguaje dominante
- Python
- Estrellas
- 12
- Forks
- 1
- Merge medio
- 1 d 1 h
- PR fusionados (30 d)
- 10
Descripción
Split out of #48 during the fair multi-queue scheduling change (`codingjoe-fair-multi-queue-scheduling`).
`defer: Silent task loss when a ready/deferred set member's task hash is gone (acquire.lua drops the id, mover.lua drops a due id whose score is gone) with no result row written. Separate durability bug; the sets have no TTL. [threadmill/backends/lua/acquire.lua, threadmill/backends/lua/mover.lua]`
## Problem
`ZPOPMIN` removes the member before the data hash is read, so when `HGET` returns nothing (hash gone, or `cjson.decode` fails) the loop moves on and the task id is left nowhere: not in the ready set, not in the running set, and no result row is written.
`mover.lua` has the same shape: it drops a due id from the deferred set when `HGET score` returns nothing, again without a result row.
The ready and deferred sets have no TTL, while the task hash expires after `lease_ttl * 3 + result_ttl` (27 h with the defaults), so a backlog or a worker outage longer than that silently loses tasks instead of failing them. That conflicts with the project's Durability and Consistency design principles.
## Open questions
- Should the expired hash be reported as a FAILED result (with what payload, given the data is gone), or should the ready/deferred sets be bounded or refreshed instead?
- Is a synthetic result row even deserializable by `peek()` and the inspector, which expect a serialized `TaskResult`?
## QED
1. Enqueue a task.
2. Delete its hash key (`{prefix}:task:{id}`) while it stays in the ready set.
3. `acquire()` — the id vanishes with no trace.
Guía de contribución
Línea de trabajo
Empieza leyendo threadmill/backends/lua/acquire.lua y threadmill/backends/lua/mover.lua; después reproduce el caso QED eliminando un hash de tarea mientras su ID permanece encolado. Revisa cómo se serializan y consumen las filas de resultados mediante peek() y el inspector. El issue solo estará terminado cuando se haya decidido la política de fallos, las IDs de las tareas ya no se pierdan silenciosamente y la cobertura de regresión verifique el comportamiento elegido.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- lua, redis
- Área
- backend, distributed-systems
- Tipo de issue
- Error
- Dificultad
- 5/5
- Tiempo estimado
- Más de una semana
- Estado de actividad
- Activo
- Claridad
- Necesita aclaración
- Aptitud para principiantes
- 35/100