aio-libs / aio-libs/aiokafka

[QUESTION] what's the recommended way to stop a consumer without auto-committing?

Aperta
#900 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub
question
Lingua principale
Python
Stelle
1.4k
Fork
269
Merge medio
1g 1h
PR unite (30g)
6

Descrizione

[consumer.stop](https://aiokafka.readthedocs.io/en/stable/api.html#aiokafka.AIOKafkaConsumer.stop) will

> Commit last consumed message if autocommit enabled

For a consumer with `enable.auto.commit=True`, I'd like to be able to shutdown without committing in order maintain better control over failure / recovery scenarios.

Given a consumer in a group meant to run indefinitely, if we encounter an exception and need to shut down 1 consumer before the message has been processed, and we call `consumer.stop` as part of the process cleanup, the message in progress is effectively lost, since `stop` will commit.

Even without encountering an exception, if we wanted to take a node out of service, and shutdown before the message has finished processing, we should be able to shutdown without committing. The message in question could take a long time to process, so we can't just wait for it to finish before shutting down.

We can take over committing in this scenario, but that's less than ideal.

We can also just kill the process without stopping the consumer, but then you'd need to wait for session timeouts before a new consumer could get those partition assignments.

It wouldn't be enough to `.close` the coordinator either, as that also will commit during finalization.

The closest thing would be getting a reference to the coordinator and calling `maybe_leave_group` while shutting down, but getting the coordinator doesn't seem to be part of the public api, regardless. Same for doing something like:

```py
# not public
consumer._enable_auto_commit = False
consumer._coordinator._enable_auto_commit = False
await consumer.stop()
```

It seems like the cleanest would be if you could do something like `await consumer.stop(commit=False)`.

Any suggestions?

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.