aio-libs / aio-libs/aiokafka

PR #1164: fetch backoff for NotLeaderForPartitionError uses fixed delay — thundering herd risk

Aberta
#1,165 3 comentários 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
Python
Estrelas
1.4k
Forks
269
Merge médio
1d 1h
PRs com merge (30d)
6

Descrição

## Summary

PR #1164 adds a per-partition backoff when a `NotLeaderForPartitionError` is received, which is a good idea. However, the backoff duration is a **fixed value** (`self._retry_backoff`, derived from `retry_backoff_ms` with no randomisation). Under a partition leader election, every consumer that was reading from that partition receives the error at the same instant, waits the exact same `retry_backoff_ms`, and then retries at the exact same instant — a classic **thundering herd** that can overwhelm the newly elected leader or the metadata service.

## Where it happens

`aiokafka/consumer/subscription_state.py` (added by the PR):

```python
def request_fetch_backoff(self, backoff: float):
self._fetch_backoff_until = time.monotonic() + backoff # fixed, no jitter
```

Called from `aiokafka/consumer/fetcher.py`:

```python
tp_state.request_fetch_backoff(self._retry_backoff)
```

`self._retry_backoff` is set once in `Fetcher.__init__` as `retry_backoff_ms / 1000` — a constant.

## Why this matters

When a broker fails and its partitions are reassigned:
1. All consumers subscribed to those partitions get `NotLeaderForPartitionError` at roughly the same time.
2. With a fixed 100 ms backoff (the default), they all sleep for exactly 100 ms and retry simultaneously.
3. The new leader receives a burst of fetch requests all at once, which can delay recovery or cause a cascade.

This pattern is described in the [AWS Architecture Blog — Exponential Backoff and Jitter](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/) (Marc Brooker). The recommended fix is **full jitter**: `sleep = random.uniform(0, base_backoff)`, or at minimum **equal jitter**: `sleep = base_backoff / 2 + random.uniform(0, base_backoff / 2)`.

## Suggested fix

One-line change in `request_fetch_backoff`:

```python
import random

def request_fetch_backoff(self, backoff: float):
jittered = backoff * random.uniform(0.5, 1.5) # ±50% spread
self._fetch_backoff_until = time.monotonic() + jittered
```

This spreads retries across a window of `[0.5×, 1.5×] retry_backoff_ms` so no two consumers retry at exactly the same millisecond.

## How this was found

This issue was identified by **[Quorum](https://github.com/KaustubhUp025/quorum)**, an open-source Gemini-powered agent that reviews merge requests for distributed coordination anti-patterns (thundering herds, missing saga compensation, lost updates, transactional outbox violations, and more). It uses GitHub's code search to verify findings across the full repository before reporting.

---

Happy to open a follow-up PR with the jitter fix if that would be helpful.

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.