apalis-dev / apalis-dev/apalis-redis

Worker pool silently dies under load (500ms response-timeout → fatal StreamError) and on restart (register_worker self-re-registration guard)

Đang mở
#76 3 bình luận 0 reaction 1 người được giao Được @geofmureithi nhận Xem trên GitHub
Ngôn ngữ chính
Rust
Star
14
Fork
5
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

## Summary

Running a `Monitor` with N `RedisStorage`-backed workers, the **entire worker pool stops on its own** under two independent conditions, both rooted in `apalis-redis`. After the workers stop, `Monitor::run_all_workers` calls `start_shutdown()`, so the process keeps running with a **silently dead** job pipeline — no panic, no shutdown signal, every worker just emits `Event::Stop`.

Versions: `apalis-core 1.0.0-rc.9`, `apalis-redis 1.0.0-rc.8`, `redis 1.2.x`.

## Trigger 1 — 500 ms default response timeout becomes a fatal `StreamError` under load

`apalis_redis::connect()` builds the manager via `Client::get_connection_manager()`, which uses redis-rs's `DEFAULT_RESPONSE_TIMEOUT = Some(500ms)`. A pool multiplexes roughly `workers × concurrency` commands over the one `ConnectionManager`; under load a single redis stall > 500 ms makes the in-flight poll/fetch return `io::ErrorKind::TimedOut`. `apalis-core` maps that to `WorkerError::StreamError`, which is **fatal** — `run_all_workers` has no retry for a stream error, so the worker future returns `Err`. Because several in-flight commands stall together, the whole pool dies at once:

```
ExitError(MonitoredErrors:
- Worker `q0`: Failed to consume task stream: timed out
- Worker `q1`: Failed to consume task stream: timed out
...
)
```

Reproduce deterministically by constructing the manager with a small `response_timeout` (e.g. 50 ms) and pushing a steady load — the pool dies within seconds-to-minutes.

Two issues here: (a) `connect()` gives no way to set/disable the response timeout, and a 500 ms per-command default is surprising for a job backend; (b) a *transient* poll error terminates the worker instead of being retried.

## Trigger 2 — `register_worker.lua` rejects self-re-registration → restart/​respawn dies instantly

`lua/register_worker.lua` raises `error("worker is still active within threshold")` when `now - last_seen < threshold`, with `threshold == keep_alive`. The steady-state heartbeat can't trip it (its sleep is never early, so `now - last_seen ≥ threshold`), but a **process restart** or a **`Monitor` restart** re-registers the same stable worker names while the previous entries are still fresh (< `keep_alive` old). The Lua `error()` surfaces as a fatal worker-stream failure, so the restarted pool dies on its first poll.

Reproduce deterministically: start the pool, kill it, restart within `keep_alive` → all workers `Event::Stop` ~1 ms after start.

A worker refreshing **its own** liveness should never be rejected — it's a keep-alive, not a duplicate claim.

## Suggested fixes

1. Let `connect()` / `RedisStorage` configure the `ConnectionManager` `response_timeout` (and consider defaulting it off for the backend); and/or have `apalis-core` retry the poll on a transient backend error rather than terminating the worker.
2. In `register_worker.lua`, allow a worker to refresh its own `last_seen` (only reject a genuinely foreign claim, if at all).

## Workaround

Vendored `apalis-redis` with `connect()` setting `response_timeout = None` and `register_worker.lua` dropping the self-re-registration guard; plus a supervisor that respawns the `Monitor` on an unexpected stop. Soak: > 30 min under load with zero deaths.

Happy to send a PR for either fix.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.