apalis-dev / apalis-dev/apalis-redis
Worker pool silently dies under load (500ms response-timeout → fatal StreamError) and on restart (register_worker self-re-registration guard)
- 主要言語
- Rust
- スター
- 14
- フォーク
- 5
- PR マージ指標
- 30日以内にマージされた PR はありません
説明
## Summary
Running a `Monitor` with N `RedisStorage`-backed workers, the **entire worker pool stops on its own** under two independent conditions, both rooted in `apalis-redis`. After the workers stop, `Monitor::run_all_workers` calls `start_shutdown()`, so the process keeps running with a **silently dead** job pipeline — no panic, no shutdown signal, every worker just emits `Event::Stop`.
Versions: `apalis-core 1.0.0-rc.9`, `apalis-redis 1.0.0-rc.8`, `redis 1.2.x`.
## Trigger 1 — 500 ms default response timeout becomes a fatal `StreamError` under load
`apalis_redis::connect()` builds the manager via `Client::get_connection_manager()`, which uses redis-rs's `DEFAULT_RESPONSE_TIMEOUT = Some(500ms)`. A pool multiplexes roughly `workers × concurrency` commands over the one `ConnectionManager`; under load a single redis stall > 500 ms makes the in-flight poll/fetch return `io::ErrorKind::TimedOut`. `apalis-core` maps that to `WorkerError::StreamError`, which is **fatal** — `run_all_workers` has no retry for a stream error, so the worker future returns `Err`. Because several in-flight commands stall together, the whole pool dies at once:
```
ExitError(MonitoredErrors:
- Worker `q0`: Failed to consume task stream: timed out
- Worker `q1`: Failed to consume task stream: timed out
...
)
```
Reproduce deterministically by constructing the manager with a small `response_timeout` (e.g. 50 ms) and pushing a steady load — the pool dies within seconds-to-minutes.
Two issues here: (a) `connect()` gives no way to set/disable the response timeout, and a 500 ms per-command default is surprising for a job backend; (b) a *transient* poll error terminates the worker instead of being retried.
## Trigger 2 — `register_worker.lua` rejects self-re-registration → restart/respawn dies instantly
`lua/register_worker.lua` raises `error("worker is still active within threshold")` when `now - last_seen < threshold`, with `threshold == keep_alive`. The steady-state heartbeat can't trip it (its sleep is never early, so `now - last_seen ≥ threshold`), but a **process restart** or a **`Monitor` restart** re-registers the same stable worker names while the previous entries are still fresh (< `keep_alive` old). The Lua `error()` surfaces as a fatal worker-stream failure, so the restarted pool dies on its first poll.
Reproduce deterministically: start the pool, kill it, restart within `keep_alive` → all workers `Event::Stop` ~1 ms after start.
A worker refreshing **its own** liveness should never be rejected — it's a keep-alive, not a duplicate claim.
## Suggested fixes
1. Let `connect()` / `RedisStorage` configure the `ConnectionManager` `response_timeout` (and consider defaulting it off for the backend); and/or have `apalis-core` retry the poll on a transient backend error rather than terminating the worker.
2. In `register_worker.lua`, allow a worker to refresh its own `last_seen` (only reject a genuinely foreign claim, if at all).
## Workaround
Vendored `apalis-redis` with `connect()` setting `response_timeout = None` and `register_worker.lua` dropping the self-re-registration guard; plus a supervisor that respawns the `Monitor` on an unexpected stop. Soak: > 30 min under load with zero deaths.
Happy to send a PR for either fix.
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
評価
この issue はまだ評価されていません。