aio-libs / aio-libs/aiokafka

PR #1164: fetch backoff for NotLeaderForPartitionError uses fixed delay — thundering herd risk

Offen
#1,165 3 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
1.4k
Forks
269
Ø Merge
1 T. 1 Std.
Gemergte PRs (30 T.)
6

Beschreibung

## Summary

PR #1164 adds a per-partition backoff when a `NotLeaderForPartitionError` is received, which is a good idea. However, the backoff duration is a **fixed value** (`self._retry_backoff`, derived from `retry_backoff_ms` with no randomisation). Under a partition leader election, every consumer that was reading from that partition receives the error at the same instant, waits the exact same `retry_backoff_ms`, and then retries at the exact same instant — a classic **thundering herd** that can overwhelm the newly elected leader or the metadata service.

## Where it happens

`aiokafka/consumer/subscription_state.py` (added by the PR):

```python
def request_fetch_backoff(self, backoff: float):
self._fetch_backoff_until = time.monotonic() + backoff # fixed, no jitter
```

Called from `aiokafka/consumer/fetcher.py`:

```python
tp_state.request_fetch_backoff(self._retry_backoff)
```

`self._retry_backoff` is set once in `Fetcher.__init__` as `retry_backoff_ms / 1000` — a constant.

## Why this matters

When a broker fails and its partitions are reassigned:
1. All consumers subscribed to those partitions get `NotLeaderForPartitionError` at roughly the same time.
2. With a fixed 100 ms backoff (the default), they all sleep for exactly 100 ms and retry simultaneously.
3. The new leader receives a burst of fetch requests all at once, which can delay recovery or cause a cascade.

This pattern is described in the [AWS Architecture Blog — Exponential Backoff and Jitter](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/) (Marc Brooker). The recommended fix is **full jitter**: `sleep = random.uniform(0, base_backoff)`, or at minimum **equal jitter**: `sleep = base_backoff / 2 + random.uniform(0, base_backoff / 2)`.

## Suggested fix

One-line change in `request_fetch_backoff`:

```python
import random

def request_fetch_backoff(self, backoff: float):
jittered = backoff * random.uniform(0.5, 1.5) # ±50% spread
self._fetch_backoff_until = time.monotonic() + jittered
```

This spreads retries across a window of `[0.5×, 1.5×] retry_backoff_ms` so no two consumers retry at exactly the same millisecond.

## How this was found

This issue was identified by **[Quorum](https://github.com/KaustubhUp025/quorum)**, an open-source Gemini-powered agent that reviews merge requests for distributed coordination anti-patterns (thundering herds, missing saga compensation, lost updates, transactional outbox violations, and more). It uses GitHub's code search to verify findings across the full repository before reporting.

---

Happy to open a follow-up PR with the jitter fix if that would be helpful.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.