cloudflare / cloudflare/pingora
Zero-weight backend panics KetamaHashing and kills the load balancer's background update task
- Dominant language
- Rust
- Stars
- 27.4k
- Forks
- 1.7k
- Avg merge
- 6h 22m
- Merged PRs (30d)
- 3
Description
A backend with weight 0 reaching `KetamaHashing` permanently kills the load balancer's background update task (or aborts the process under `panic = "abort"`).
## Where
- `pingora-load-balancing/src/selection/consistent.rs`, `KetamaHashing::build_with_config`, passes `b.weight as u32` straight into `pingora_ketama::Bucket::new`.
- `pingora-ketama/src/lib.rs`, `Bucket::new`: `assert!(weight != 0, "weight must be at least one")`.
## How it triggers
`Backend::weight` is a plain, unvalidated `usize` (`pingora-load-balancing/src/lib.rs`). Neither `Backend::new_with_weight` nor the `ServiceDiscovery` trait rejects 0. Some discovery backends use weight 0 as a draining/disabled marker, and a typo in a static config or DNS SRV record produces the same thing. Once one such backend is in the `BTreeSet` passed to `KetamaHashing::build_with_config`, `Bucket::new` panics.
That call happens inside `LoadBalancer::update()` (`pingora-load-balancing/src/lib.rs`), driven by the background loop in `pingora-load-balancing/src/background.rs` (`LoadBalancer::run`). Nothing wraps it in `catch_unwind` or `spawn_blocking`. The newer `LoadBalancerGroup` rebuild path already runs `build_selector` inside `spawn_blocking` and checks `JoinError::is_panic()`, so this class of panic is handled there, just not on the plain `LoadBalancer` path.
## Impact
Once the panic fires, the spawned update task ends and is never restarted:
- Backend membership changes (new/removed hosts) stop being applied.
- Health check state freezes at whatever it was.
- Under `panic = "abort"`, the whole process goes down instead.
All from a single zero-weight entry in one discovery response, when the consumer uses consistent hashing (a common choice for cache or session-sticky routing).
## Suggested fix
Clamp the weight to a minimum of 1 at the `Backend` to `Bucket` boundary in `KetamaHashing::build_with_config`, since `Backend::weight` is untrusted, unvalidated input and `Bucket` requires a positive weight by contract.
Contributor guide
Research direction
Start in pingora-load-balancing/src/selection/consistent.rs at KetamaHashing::build_with_config, then read pingora-ketama/src/lib.rs and Backend in pingora-load-balancing/src/lib.rs to confirm the weight contract. Check the update flow through LoadBalancer::update and pingora-load-balancing/src/background.rs::LoadBalancer::run. Done means a zero-weight backend no longer panics during consistent-hashing updates and the relevant load-balancing tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100