confluentinc / confluentinc/confluent-kafka-python

Producer goes to retry-backoff by send_offsets_to_transaction() thus degrading performance

Open
#967 6 comments 0 reactions 0 assignees View on GitHub
component:librdkafka component:producer investigate further
Dominant language
Python
Stars
509
Forks
964
Avg merge
2d 2h
Merged PRs (30d)
14

Description

Description
===========
For some reason the `send_offsets_to_transaction()` call will be 50+ times slower when there is more than one message in the input topic of the transactor.

How to reproduce
================
I've made a [reproduction demo](https://github.com/Traktormaster/kafka-tr-issue-201011) that has two modes of operation. In one the issue is present and in the other it is absent.

The [readme](https://github.com/Traktormaster/kafka-tr-issue-201011/blob/main/README.md) in that repository contains detailed information about the environment, the issue and the demo. There are log excerpts from the clients and the broker.

Checklist
=========
Please provide the following information:

- [x] confluent-kafka-python and librdkafka version (`confluent_kafka.version()` and `confluent_kafka.libversion()`): ('1.5.0', 17104896) ('1.5.0', 17105151)
- [x] Apache Kafka broker version: 2.5.0 or 2.6.0
- [x] Client configuration: *see reproduction demo*
- [x] Operating system: Linux 5.8.14
- [x] Provide client logs (with `'debug': '..'` as necessary): *see reproduction demo*
- [x] Provide broker log excerpts: *see reproduction demo*
- [x] Critical issue: nearly unusable performance when the issue is present

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.