confluentinc / confluentinc/librdkafka

The connection from producer to Kafka cluster is flapping

Open
#5,000 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
1k
Forks
3.3k
Avg merge
5d 12h
Merged PRs (30d)
6

Description

The Kafka is deployed in a kubernetes environment with 3 pods in a kafka cluster.

The producer kafka connection keeps flapping. The error callback is producing below low -

**err: Local: Broker transport failure

reason: kafka-0.kafka-headless.common.svc.cluster.local:9092/1001: Receive failed: Disconnected**

What is the reason for this flap?

I have below config set, but i am not getting any additional information.

**rd_kafka_conf_set(conf, "debug", "broker,topic,msg", NULL, 0);**
**rd_kafka_conf_set(conf, "log_level", "7", NULL, 0);**

Other config:

rd_kafka_conf_set(conf, "socket.keepalive.enable", "true", NULL, 0);
rd_kafka_conf_set(conf, "log.connection.close", "true",NULL, 0);

Contributor guide

Open the contributing guide

Research direction

Start with the reported rd_kafka_conf_set debug, log_level, socket.keepalive.enable, and log.connection.close settings, then inspect the broker and Kubernetes logs for kafka-0.kafka-headless.common.svc.cluster.local:9092. Done means identifying a reproducible cause for the disconnects and documenting the log evidence or configuration change that prevents the connection from flapping.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, kubernetes
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.