confluentinc / confluentinc/confluent-kafka-python
Different per-topic timeout configuration
- Dominant language
- Python
- Stars
- 509
- Forks
- 964
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 14
Description
Description
===========
I'm looking to use a single Kafka producer for two slightly differnt use cases/message types. One that is just fire-and-forget and that will simply enqueue a message without wait for any acknowledgment.
For the second use case I'd like to make sure that the message was written to broker before returning a response - similar to what's described in [#1174](https://github.com/confluentinc/librdkafka/issues/1174)
It seems that the way to achieve this would be to set a low message timeout and wait for `poll()` to acknowledge message delivery via the callback.
Having a low global message timeout wouldn't be very good for the fire-and-forget use cause, so it seems that the best option would be to use the per-topic configuration.
Was there any progress on implementing this since https://github.com/confluentinc/confluent-kafka-python/issues/328#issuecomment-371199811 ? It looks like none of the APIs are exposed through the python client.
If there's still no way to provide per-topic config I'd be open to work on it and make a pull request, but would like to understand what the preferred approach is. maybe exposing the topic creation methods on the produce with a config dict would be enough?
Thanks in advance
Checklist
=========
Please provide the following information:
- [X] confluent-kafka-python==2.20 librdkafka==2.2.0
- [X] Apache Kafka broker version: 3.4.0
- [ ] Client configuration: `{...}`
- [ ] Operating system:
- [ ] Provide client logs (with `'debug': '..'` as necessary)
- [ ] Provide broker log excerpts
- [ ] Critical issue
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.