confluentinc / confluentinc/confluent-kafka-python

Producer logging thousands of errors in a very short timespan (Broker transport failure)

Open
#1,328 8 comments 0 reactions 0 assignees View on GitHub
bug component:librdkafka investigate further priority:low
Dominant language
Python
Stars
509
Forks
964
Avg merge
2d 2h
Merged PRs (30d)
14

Description

Description
===========
I have a problem with a producer in an API. Specifically, once in a while it'll throw errors like `SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)`, which wouldn't be a problem on its own if it was a few times, but the problem is that whenever it happens that there's a transport failure I will get THOUSANDS of log entries, like it's retrying it every single millisecond.

For instance, here's 1½ minute where it logged that error 10 000 times:

![image](https://user-images.githubusercontent.com/202696/165065427-b4572ccc-e352-468d-9b02-079714c47e29.png)

I'm fine with it logging errors and retrying the request, but I don't understand why it's retrying it so frequently. I've tried fiddling with producer config to make it back off, but even with the defaults it seems like it shouldn't really be doing it that often.

Here's an example of the error:

```
FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker:9092/bootstrap]: sasl_ssl://broker:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)
```

My producing code looks like this:

```
def produce_event(
topic, event_data, kafka_producer
):
"""Produce an Event to Kafka."""
event_id = str(uuid.uuid4())
event_dt = datetime.datetime.utcnow()
event = Event(
event_id=UUID(value=event_id),
event_datetime=Timestamp(seconds=int(event_dt.timestamp()), nanos=0),
event_data=JsonString(content=json.dumps(event_data)),
)
kafka_producer.produce(topic, key=event_id, value=event, on_delivery=kafka_delivery_report)
kafka_producer.poll()
```

How to reproduce
================

Unknown

Checklist
=========
Please provide the following information:

- [x] confluent-kafka-python and librdkafka version: ('1.8.2', 17302016) and ('1.8.2', 17302271)
- [x] Apache Kafka broker version: 2.7.0 (on Confluent Platform 6.1.4)
- [x] Client configuration: `{
"retry.backoff.ms": 1000,
"reconnect.backoff.ms": 500,
"reconnect.backoff.max.ms": 5000,
"key.serializer": string_serializer,
"value.serializer": proto_serializer
}`
- [x] Operating system: Debian
- [x] Provide client logs (with `'debug': '..'` as necessary)
- [ ] Provide broker log excerpts
- [ ] Critical issue

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.