PR #1164: fetch backoff for NotLeaderForPartitionError uses fixed delay — thundering herd risk
- Vorherrschende Sprache
- Python
- Sterne
- 1.4k
- Forks
- 269
- Ø Merge
- 1 T. 1 Std.
- Gemergte PRs (30 T.)
- 6
Beschreibung
## Summary
PR #1164 adds a per-partition backoff when a `NotLeaderForPartitionError` is received, which is a good idea. However, the backoff duration is a **fixed value** (`self._retry_backoff`, derived from `retry_backoff_ms` with no randomisation). Under a partition leader election, every consumer that was reading from that partition receives the error at the same instant, waits the exact same `retry_backoff_ms`, and then retries at the exact same instant — a classic **thundering herd** that can overwhelm the newly elected leader or the metadata service.
## Where it happens
`aiokafka/consumer/subscription_state.py` (added by the PR):
```python
def request_fetch_backoff(self, backoff: float):
self._fetch_backoff_until = time.monotonic() + backoff # fixed, no jitter
```
Called from `aiokafka/consumer/fetcher.py`:
```python
tp_state.request_fetch_backoff(self._retry_backoff)
```
`self._retry_backoff` is set once in `Fetcher.__init__` as `retry_backoff_ms / 1000` — a constant.
## Why this matters
When a broker fails and its partitions are reassigned:
1. All consumers subscribed to those partitions get `NotLeaderForPartitionError` at roughly the same time.
2. With a fixed 100 ms backoff (the default), they all sleep for exactly 100 ms and retry simultaneously.
3. The new leader receives a burst of fetch requests all at once, which can delay recovery or cause a cascade.
This pattern is described in the [AWS Architecture Blog — Exponential Backoff and Jitter](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/) (Marc Brooker). The recommended fix is **full jitter**: `sleep = random.uniform(0, base_backoff)`, or at minimum **equal jitter**: `sleep = base_backoff / 2 + random.uniform(0, base_backoff / 2)`.
## Suggested fix
One-line change in `request_fetch_backoff`:
```python
import random
def request_fetch_backoff(self, backoff: float):
jittered = backoff * random.uniform(0.5, 1.5) # ±50% spread
self._fetch_backoff_until = time.monotonic() + jittered
```
This spreads retries across a window of `[0.5×, 1.5×] retry_backoff_ms` so no two consumers retry at exactly the same millisecond.
## How this was found
This issue was identified by **[Quorum](https://github.com/KaustubhUp025/quorum)**, an open-source Gemini-powered agent that reviews merge requests for distributed coordination anti-patterns (thundering herds, missing saga compensation, lost updates, transactional outbox violations, and more). It uses GitHub's code search to verify findings across the full repository before reporting.
---
Happy to open a follow-up PR with the jitter fix if that would be helpful.
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.