kafka sink accumulates sarama Broker responseReceiver goroutines as broker connections expand

Background

We are seeing the goroutine count continuously increase on one TiCDC node for more than a day in a customer environment. This does not happen on every cluster.

I correlated two goroutine dumps (around 2026-03-20 19:00 and 2026-03-20 20:00 UTC) with the node-local TiCDC log from the same node.

Raw goroutine dumps:

• see files

What grows

Between the two dumps:

• total goroutines: 23584 -> 23602 (+18)
• github.com/IBM/sarama.(*Broker).responseReceiver: 914 -> 932 (+18)
• logpuller worker goroutines stayed flat:
  • logservice/logpuller.(*regionRequestWorker).receiveAndDispatchChangeEvents: 1824 -> 1824
  • logservice/logpuller.(*regionRequestWorker).processRegionSendTask: 1824 -> 1824

So the growth is concentrated in Sarama broker receiver goroutines, not in logpuller workers.

Correlated log evidence from the same node

In the 2026-03-20 19:* window:

• Connected to broker: 24
• Closed connection to broker: 7
• net broker connection growth: +17

This matches the responseReceiver goroutine delta (+18) very closely.

I did not find sink close/rebuild logs in the 19:* or 20:* window:

• no Sink manager closing sink factory
• no Close kafka async producer client success
• no Close kafka async producer success
• no Kafka DDL producer closed
• no async producer exit since context is done

So in this one-hour window, this does not currently look like sink recreation.

Observed pattern

The Sarama logs show periodic metadata refresh batches roughly every 9 minutes:

• 19:07 / 19:16 / 19:25 / 19:34 / 19:43 / 19:52
• 20:01 / 20:10 / 20:19 / 20:28 / 20:37

During some refreshes we see broken pipes, deregistration, and immediate re-registration of brokers in the same refresh cycle.

Example around 19:36:24:

• brokers #91, #8, #50 hit write: broken pipe
• the connections are closed and deregistered
• the same brokers are registered again immediately in that refresh cycle

Example around 20:12:24:

• brokers #59, #64, #44, #2 hit write: broken pipe
• the connections are closed and deregistered
• the same brokers are registered again immediately in that refresh cycle

Relevant code paths

TiCDC Kafka sink creates three long-lived Sarama clients per sink:

• admin client: pkg/sink/kafka/sarama_factory.go
• sync producer client: pkg/sink/kafka/sarama_factory.go
• async producer client: pkg/sink/kafka/sarama_factory.go

TiCDC also configures and drives them like this:

• pkg/sink/kafka/sarama_config.go: config.Metadata.RefreshFrequency = 9 * time.Minute
• downstreamadapter/sink/kafka/sink.go: DML producer heartbeat every 5s
• downstreamadapter/sink/kafka/sink.go: DDL producer heartbeat every 5s
• downstreamadapter/sink/topicmanager/kafka_topic_manager.go: admin heartbeat every 5s
• all three Heartbeat() implementations iterate over client.Brokers() and call ApiVersions(...)

On the Sarama side:

• client.LeastLoadedBroker() opens the selected broker
• Broker.Open() starts a responseReceiver goroutine

Suspected problem

From the operator side this looks like a goroutine leak, but the current evidence suggests a narrower issue:

• long-lived Kafka clients gradually expand the set of connected brokers
• once a broker connection is opened, the 5-second heartbeat keeps it active
• responseReceiver goroutines therefore grow monotonically until the per-client broker set reaches a plateau

So I am not claiming that this is already proven to be an infinite leak. The current claim is narrower:

• the goroutine growth is real
• it is driven by Kafka/Sarama broker connection growth inside long-lived clients
• it is not explained by sink recreation in the analyzed one-hour window
• the steady-state goroutine count can still become very high and look like a leak operationally

Questions

Is this considered expected behavior, or should TiCDC avoid keeping so many broker connections alive per sink/client?

Possible directions that seem relevant:

• reduce the heartbeat scope
• allow more broker connections to age out instead of keeping them alive every 5s
• avoid expanding the broker connection set so aggressively during metadata refresh
• provide a documented limit/workaround for large-broker-count Kafka clusters

Start with pkg/sink/kafka/sarama_factory.go and pkg/sink/kafka/sarama_config.go, then trace the heartbeat paths in downstreamadapter/sink/kafka/sink.go and downstreamadapter/sink/topicmanager/kafka_topic_manager.go into Sarama's Broker.Open and responseReceiver. Compare broker connection and goroutine behavior across metadata refreshes; done requires an agreed expected connection policy and evidence that the selected behavior is bounded or otherwise documented.