aio-libs / aio-libs/aiokafka

[QUESTION] Does aiokafka have an equivalent to `enable.auto.offset.store=false` and `store()`

Offen
#826 0 Kommentare 3 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
question
Vorherrschende Sprache
Python
Sterne
1.4k
Forks
269
Ø Merge
1 T. 1 Std.
Gemergte PRs (30 T.)
6

Beschreibung

Calling `getmany` seems to [suggest it updates consumed offsets at the same time](https://github.com/aio-libs/aiokafka/blob/081087d1c92e2bf6ee3660843f641bfb4f3fd754/aiokafka/consumer/fetcher.py#L1043-L1045).

There are warnings in various places around manual commits, and suggestions that committing after each message is slow. Ex, from: https://github.com/edenhill/librdkafka/wiki/FAQ#why-committing-each-message-is-slow

> There are actually two layers of offset commit, first there is the offset store (offset_store(), store_offsets(), etc) that locally stores (in memory) the offset to be committed, then there's the actual commit which commits the stored offset to the broker. **An alternative is thus to explicitly store() each message's offset after processing but then relying on the auto offset committer to actually commit the offsets for you**. Use enable.auto.offset.store=false to disable automatic offset store (which is by default done prior to your processing).

Is there anywhere in aiokafka that allows you to individually control the offsets that are eligible to be committed, while still using auto commit, similar to suggested above in that librdkafka FAQ -- or is manual commit the only option?

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.