aio-libs / aio-libs/aiokafka

Change the implementation of lingering to properly handle retries and force send if batch is full/closed

Ouverte
#449 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
priority:normal
Langage dominant
Python
Étoiles
1.4k
Forks
269
Merge moyen
1 j 1 h
PR mergées (30 j)
6

Description

Currently we send the first message ASAP from the producer. This makes code like this less efficient:

```
for i in range(10):
await producer.send(topic, msg=str(i).encode())
```

There are 2 related problems:
* Batches with linger_ms set will only force a longer wait period, but check if next batch is full. We need to change that so that lingering can be skipped if next batch is ready for send.
* If we don't use lingering there is no guarantee on how batches are produced. While this is not a big problem the code above may have non-constant performance. I would still have at least some basic lingering as a default setting to be sure batches are not sent right away if we have a busy loop.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.