confluentinc / confluentinc/confluent-kafka-python

Different per-topic timeout configuration

Open
#1,671 2 comments 0 reactions 0 assignees View on GitHub
api-change component:producer priority:high
Dominant language
Python
Stars
509
Forks
964
Avg merge
2d 2h
Merged PRs (30d)
14

Description

Description
===========

I'm looking to use a single Kafka producer for two slightly differnt use cases/message types. One that is just fire-and-forget and that will simply enqueue a message without wait for any acknowledgment.
For the second use case I'd like to make sure that the message was written to broker before returning a response - similar to what's described in [#1174](https://github.com/confluentinc/librdkafka/issues/1174)

It seems that the way to achieve this would be to set a low message timeout and wait for `poll()` to acknowledge message delivery via the callback.

Having a low global message timeout wouldn't be very good for the fire-and-forget use cause, so it seems that the best option would be to use the per-topic configuration.
Was there any progress on implementing this since https://github.com/confluentinc/confluent-kafka-python/issues/328#issuecomment-371199811 ? It looks like none of the APIs are exposed through the python client.

If there's still no way to provide per-topic config I'd be open to work on it and make a pull request, but would like to understand what the preferred approach is. maybe exposing the topic creation methods on the produce with a config dict would be enough?

Thanks in advance

Checklist
=========
Please provide the following information:

- [X] confluent-kafka-python==2.20 librdkafka==2.2.0
- [X] Apache Kafka broker version: 3.4.0
- [ ] Client configuration: `{...}`
- [ ] Operating system:
- [ ] Provide client logs (with `'debug': '..'` as necessary)
- [ ] Provide broker log excerpts
- [ ] Critical issue

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.