Box Redis uses allkeys-lru for the task store, and threadmill drops evicted tasks silently
- Dominant language
- Python
- Stars
- 4
- Forks
- 0
- Avg merge
- 8h 24m
- Merged PRs (30d)
- 115
Description
## Problem
The Django cache and the task store share one Redis instance, and that instance
is allowed to evict task data.
- `root/settings.py:206` derives the cache from `REDIS_URL`:
`CACHES = {"default": env.cache("REDIS_URL", default="locmemcache://")}`.
- `root/settings.py:385` gives the task backend the same `REDIS_URL`.
- On the box both land on the Redis from `oci://ghcr.io/codingjoe/the-box-redis:main`,
started with `--requirepass`, `--maxmemory 384mb`, `--maxmemory-policy allkeys-lru`.
`allkeys-lru` lets Redis evict any key, including `threadmill:task:` hashes,
and threadmill's `acquire.lua` treats a missing hash as an empty queue. The script
pops the id first, then reads the data:
```lua
local result = redis.call('ZPOPMIN', KEYS[i * 2])
if #result > 0 then
local task_id = result[1]
local data = redis.call('HGET', ARGV[3] .. task_id, 'data')
if data then
...
end
end
```
If `data` is nil the loop moves on and the script returns nil. The task id has
already been removed from the queue, so the task is gone: no error, no failed
result, no retry callback. The spam-scan policy that is meant to retry until it
succeeds cannot see it either, because the task never comes back to the executor.
## Impact
Silent, untraceable message loss under memory pressure, which is also when the
queue is busiest: unscanned messages, outgoing mail that is never delivered,
webhooks that are never dispatched. Nothing in the task store records that the
work was expected, so no alert fires and no reconciliation can find it. Cache
keys and task keys compete for the same 384 MB, and the cache is the side that
is designed to be eviction-friendly.
## Suggested fix
Points to decide:
- Separate the task store from the cache. A separate logical DB on the same
instance is not enough, because `maxmemory-policy` is per instance, so a
cache-heavy DB still evicts task keys.
- Give the task store an eviction policy that fails loudly rather than dropping
data (`noeviction`, or `volatile-*` if the policy can be kept off task keys).
- Revisit the budget. 384 MB covers scans, deliveries and webhook retry ZSETs at
the same time, and the retry schedule added here defers runs for hours.
- Upstream in threadmill: `acquire.lua` should record a failed result when a
popped task id has no hash, instead of skipping it silently.
## Verification
Read `threadmill/backends/lua/acquire.lua` from the installed 0.7.1
(`.venv/lib/python3.14/site-packages/threadmill/backends/lua/acquire.lua:18-43`).
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.