codingjoe / codingjoe/relay

Box Redis uses allkeys-lru for the task store, and threadmill drops evicted tasks silently

Open
#227 0 comments 0 reactions 0 assignees View on GitHub
hardening
Dominant language
Python
Stars
4
Forks
0
Avg merge
8h 24m
Merged PRs (30d)
115

Description

## Problem

The Django cache and the task store share one Redis instance, and that instance
is allowed to evict task data.

- `root/settings.py:206` derives the cache from `REDIS_URL`:
`CACHES = {"default": env.cache("REDIS_URL", default="locmemcache://")}`.
- `root/settings.py:385` gives the task backend the same `REDIS_URL`.
- On the box both land on the Redis from `oci://ghcr.io/codingjoe/the-box-redis:main`,
started with `--requirepass`, `--maxmemory 384mb`, `--maxmemory-policy allkeys-lru`.

`allkeys-lru` lets Redis evict any key, including `threadmill:task:` hashes,
and threadmill's `acquire.lua` treats a missing hash as an empty queue. The script
pops the id first, then reads the data:

```lua
local result = redis.call('ZPOPMIN', KEYS[i * 2])
if #result > 0 then
local task_id = result[1]
local data = redis.call('HGET', ARGV[3] .. task_id, 'data')
if data then
...
end
end
```

If `data` is nil the loop moves on and the script returns nil. The task id has
already been removed from the queue, so the task is gone: no error, no failed
result, no retry callback. The spam-scan policy that is meant to retry until it
succeeds cannot see it either, because the task never comes back to the executor.

## Impact

Silent, untraceable message loss under memory pressure, which is also when the
queue is busiest: unscanned messages, outgoing mail that is never delivered,
webhooks that are never dispatched. Nothing in the task store records that the
work was expected, so no alert fires and no reconciliation can find it. Cache
keys and task keys compete for the same 384 MB, and the cache is the side that
is designed to be eviction-friendly.

## Suggested fix

Points to decide:

- Separate the task store from the cache. A separate logical DB on the same
instance is not enough, because `maxmemory-policy` is per instance, so a
cache-heavy DB still evicts task keys.
- Give the task store an eviction policy that fails loudly rather than dropping
data (`noeviction`, or `volatile-*` if the policy can be kept off task keys).
- Revisit the budget. 384 MB covers scans, deliveries and webhook retry ZSETs at
the same time, and the retry schedule added here defers runs for hours.
- Upstream in threadmill: `acquire.lua` should record a failed result when a
popped task id has no hash, instead of skipping it silently.

## Verification

Read `threadmill/backends/lua/acquire.lua` from the installed 0.7.1
(`.venv/lib/python3.14/site-packages/threadmill/backends/lua/acquire.lua:18-43`).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.