[QUESTION] what's the recommended way to stop a consumer without auto-committing?
- Lingua principale
- Python
- Stelle
- 1.4k
- Fork
- 269
- Merge medio
- 1g 1h
- PR unite (30g)
- 6
Descrizione
[consumer.stop](https://aiokafka.readthedocs.io/en/stable/api.html#aiokafka.AIOKafkaConsumer.stop) will
> Commit last consumed message if autocommit enabled
For a consumer with `enable.auto.commit=True`, I'd like to be able to shutdown without committing in order maintain better control over failure / recovery scenarios.
Given a consumer in a group meant to run indefinitely, if we encounter an exception and need to shut down 1 consumer before the message has been processed, and we call `consumer.stop` as part of the process cleanup, the message in progress is effectively lost, since `stop` will commit.
Even without encountering an exception, if we wanted to take a node out of service, and shutdown before the message has finished processing, we should be able to shutdown without committing. The message in question could take a long time to process, so we can't just wait for it to finish before shutting down.
We can take over committing in this scenario, but that's less than ideal.
We can also just kill the process without stopping the consumer, but then you'd need to wait for session timeouts before a new consumer could get those partition assignments.
It wouldn't be enough to `.close` the coordinator either, as that also will commit during finalization.
The closest thing would be getting a reference to the coordinator and calling `maybe_leave_group` while shutting down, but getting the coordinator doesn't seem to be part of the public api, regardless. Same for doing something like:
```py
# not public
consumer._enable_auto_commit = False
consumer._coordinator._enable_auto_commit = False
await consumer.stop()
```
It seems like the cleanest would be if you could do something like `await consumer.stop(commit=False)`.
Any suggestions?
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.