cloudflare / cloudflare/pingora

Zero-weight backend panics KetamaHashing and kills the load balancer's background update task

Open Beginner friendly
#1,006 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
27.4k
Forks
1.7k
Avg merge
6h 22m
Merged PRs (30d)
3

Description

A backend with weight 0 reaching `KetamaHashing` permanently kills the load balancer's background update task (or aborts the process under `panic = "abort"`).

## Where

- `pingora-load-balancing/src/selection/consistent.rs`, `KetamaHashing::build_with_config`, passes `b.weight as u32` straight into `pingora_ketama::Bucket::new`.
- `pingora-ketama/src/lib.rs`, `Bucket::new`: `assert!(weight != 0, "weight must be at least one")`.

## How it triggers

`Backend::weight` is a plain, unvalidated `usize` (`pingora-load-balancing/src/lib.rs`). Neither `Backend::new_with_weight` nor the `ServiceDiscovery` trait rejects 0. Some discovery backends use weight 0 as a draining/disabled marker, and a typo in a static config or DNS SRV record produces the same thing. Once one such backend is in the `BTreeSet` passed to `KetamaHashing::build_with_config`, `Bucket::new` panics.

That call happens inside `LoadBalancer::update()` (`pingora-load-balancing/src/lib.rs`), driven by the background loop in `pingora-load-balancing/src/background.rs` (`LoadBalancer::run`). Nothing wraps it in `catch_unwind` or `spawn_blocking`. The newer `LoadBalancerGroup` rebuild path already runs `build_selector` inside `spawn_blocking` and checks `JoinError::is_panic()`, so this class of panic is handled there, just not on the plain `LoadBalancer` path.

## Impact

Once the panic fires, the spawned update task ends and is never restarted:
- Backend membership changes (new/removed hosts) stop being applied.
- Health check state freezes at whatever it was.
- Under `panic = "abort"`, the whole process goes down instead.

All from a single zero-weight entry in one discovery response, when the consumer uses consistent hashing (a common choice for cache or session-sticky routing).

## Suggested fix

Clamp the weight to a minimum of 1 at the `Backend` to `Bucket` boundary in `KetamaHashing::build_with_config`, since `Backend::weight` is untrusted, unvalidated input and `Bucket` requires a positive weight by contract.

Contributor guide

Open the contributing guide

Research direction

Start in pingora-load-balancing/src/selection/consistent.rs at KetamaHashing::build_with_config, then read pingora-ketama/src/lib.rs and Backend in pingora-load-balancing/src/lib.rs to confirm the weight contract. Check the update flow through LoadBalancer::update and pingora-load-balancing/src/background.rs::LoadBalancer::run. Done means a zero-weight backend no longer panics during consistent-hashing updates and the relevant load-balancing tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.