apalis-dev / apalis-dev/apalis-redis

Worker pool silently dies under load (500ms response-timeout → fatal StreamError) and on restart (register_worker self-re-registration guard)

Abierto
#76 3 comentarios 0 reacciones 1 asignado Reclamado por @geofmureithi Ver en GitHub
Lenguaje dominante
Rust
Estrellas
14
Forks
5
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

## Summary

Running a `Monitor` with N `RedisStorage`-backed workers, the **entire worker pool stops on its own** under two independent conditions, both rooted in `apalis-redis`. After the workers stop, `Monitor::run_all_workers` calls `start_shutdown()`, so the process keeps running with a **silently dead** job pipeline — no panic, no shutdown signal, every worker just emits `Event::Stop`.

Versions: `apalis-core 1.0.0-rc.9`, `apalis-redis 1.0.0-rc.8`, `redis 1.2.x`.

## Trigger 1 — 500 ms default response timeout becomes a fatal `StreamError` under load

`apalis_redis::connect()` builds the manager via `Client::get_connection_manager()`, which uses redis-rs's `DEFAULT_RESPONSE_TIMEOUT = Some(500ms)`. A pool multiplexes roughly `workers × concurrency` commands over the one `ConnectionManager`; under load a single redis stall > 500 ms makes the in-flight poll/fetch return `io::ErrorKind::TimedOut`. `apalis-core` maps that to `WorkerError::StreamError`, which is **fatal** — `run_all_workers` has no retry for a stream error, so the worker future returns `Err`. Because several in-flight commands stall together, the whole pool dies at once:

```
ExitError(MonitoredErrors:
- Worker `q0`: Failed to consume task stream: timed out
- Worker `q1`: Failed to consume task stream: timed out
...
)
```

Reproduce deterministically by constructing the manager with a small `response_timeout` (e.g. 50 ms) and pushing a steady load — the pool dies within seconds-to-minutes.

Two issues here: (a) `connect()` gives no way to set/disable the response timeout, and a 500 ms per-command default is surprising for a job backend; (b) a *transient* poll error terminates the worker instead of being retried.

## Trigger 2 — `register_worker.lua` rejects self-re-registration → restart/​respawn dies instantly

`lua/register_worker.lua` raises `error("worker is still active within threshold")` when `now - last_seen < threshold`, with `threshold == keep_alive`. The steady-state heartbeat can't trip it (its sleep is never early, so `now - last_seen ≥ threshold`), but a **process restart** or a **`Monitor` restart** re-registers the same stable worker names while the previous entries are still fresh (< `keep_alive` old). The Lua `error()` surfaces as a fatal worker-stream failure, so the restarted pool dies on its first poll.

Reproduce deterministically: start the pool, kill it, restart within `keep_alive` → all workers `Event::Stop` ~1 ms after start.

A worker refreshing **its own** liveness should never be rejected — it's a keep-alive, not a duplicate claim.

## Suggested fixes

1. Let `connect()` / `RedisStorage` configure the `ConnectionManager` `response_timeout` (and consider defaulting it off for the backend); and/or have `apalis-core` retry the poll on a transient backend error rather than terminating the worker.
2. In `register_worker.lua`, allow a worker to refresh its own `last_seen` (only reject a genuinely foreign claim, if at all).

## Workaround

Vendored `apalis-redis` with `connect()` setting `response_timeout = None` and `register_worker.lua` dropping the self-re-registration guard; plus a supervisor that respawns the `Monitor` on an unexpected stop. Soak: > 30 min under load with zero deaths.

Happy to send a PR for either fix.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.