redpanda-data / redpanda-data/connect

redpanda (kafka (eventhubs)) input stalling for exactly 15 minutes

Open
#4,648 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
8.8k
Forks
969
Avg merge
1d 13h
Merged PRs (30d)
64

Description

Hi, we're seeing occasional input stalls in our redpanda connect pods consuming eventhubs data. Crucially: logstash (which is consuming the same data in parallel while we're trying to replace it with redpanda connect) is not seeing this, so it can't be purely an eventhubs problem.

Relevant parts of our setup:

  • right before the data enters eventhubs, we timestamp it
  • at the moment we start processing it in logstash/connect we timestamp it again and compute the diff with the first timestamp to monitor processinglag
  • we also tag the data with the eventhubs partition
  • both output to the same elasticsearch in different indices, and we monitor processinglag, and connect is seeing peaks of 900s on specific partitions

enabling debug logging (see collapsible below for detailed log) shows about this:

  • happily processing, printing something with every batch sent
  • something happens like a partition rebalance, takes at most some seconds to finish
  • 15 minutes of nothing
  • happy processing again

what we've tried:

  • we used to use the kafka input at version 1.0.0 (probably an old default we copied since forever), so we switched this to the redpanda input instead, same thing happens
  • changed settings from defaults (some of these variables play/interfere with each other so have changed multiple times) but no improvement observed
    • session_timeout: default 1m → 10s
    • conn_idle_timeout: raised to 20s
    • request_timeout_overhead: raised to 40s
    • rebalance_timeout: pinned explicitly at its default 45s (not actually changed, just made explicit)
    • tcp.tcp_user_timeout: added (default 0/disabled) → 8s
    • tcp.keep_alive.idle: raised default 15s → 30s
    • unordered_processing on & off again
    • max_yield_batch_bytes: 10000000
    • fetch_max_partition_bytes: 10000000
    • partition_buffer_bytes: 12000000

again, crucially the logstash is not experiencing these stalls so it's almost certain it's connect-specific, but it's hard to debug especially since the debuglogs are showing nothing at all in that 15 minute window

Any suggestions in helping debugging or potentially relevant 15-minute timeouts (or 5-minute timeouts with 2 retries etc etc) are very welcome!

debug logs 
time="2026-07-22T01:50:51Z" level=debug msg="Successfully dispatched [3] documents in 45ms (66.666667 docs/sec)" @service=benthos label="" path=root.output.broker.outputs.0
time="2026-07-22T01:50:51Z" level=debug msg="Successfully dispatched [13] documents in 123ms (105.691057 docs/sec)" @service=benthos label="" path=root.output.broker.outputs.0
time="2026-07-22T01:50:55Z" level=debug msg="Successfully dispatched [7] documents in 12ms (583.333333 docs/sec)" @service=benthos label="" path=root.output.broker.outputs.0
time="2026-07-22T01:51:01Z" level=warning msg="Kafka broker read failed" @service=benthos addr="prodmheventhubs.servicebus.windows.net:9093" broker=0 err="context deadline exceeded" label=kafka_input path
=root.input.broker.inputs.0 request=DescribeGroups
time="2026-07-22T01:51:01Z" level=debug msg="Failed to fetch group lags: request DescribeGroups has 1 separate shard errors, first: context deadline exceeded" @service=benthos label=kafka_input path=root.
input.broker.inputs.0
time="2026-07-22T01:51:01Z" level=error msg="default commit failed" @service=benthos err="the broker connection has died and the request will be retried on a new connection" group=traffic-streams-prod-mhb
e-events-consumer-v3 label=kafka_input path=root.input.broker.inputs.0
time="2026-07-22T01:51:05Z" level=debug msg="heartbeat errored" @service=benthos err="REBALANCE_IN_PROGRESS: The group is rebalancing, so a rejoin is needed." group=traffic-streams-prod-mhbe-events-consum
er-v3 label=kafka_input path=root.input.broker.inputs.0
time="2026-07-22T01:51:05Z" level=debug msg="consumer calling onRevoke at the end of a session; consumer did not change any client-side subscription" @service=benthos group=traffic-streams-prod-mhbe-event
s-consumer-v3 label=kafka_input path=root.input.broker.inputs.0
time="2026-07-22T01:51:05Z" level=debug msg="joining group" @service=benthos group=traffic-streams-prod-mhbe-events-consumer-v3 label=kafka_input path=root.input.broker.inputs.0
time="2026-07-22T01:51:06Z" level=warning msg="Kafka broker read failed" @service=benthos addr="prodmheventhubs.servicebus.windows.net:9093" broker=-2147483648 err="context deadline exceeded" label=kafka_input path=root.input.broker.inputs.0 request=Metadata
time="2026-07-22T01:51:06Z" level=debug msg="Failed to fetch group lags: context deadline exceeded" @service=benthos label=kafka_input path=root.input.broker.inputs.0
time="2026-07-22T01:51:08Z" level=debug msg=joined @service=benthos generation=74967704 group=traffic-streams-prod-mhbe-events-consumer-v3 instance_id="<nil>" label=kafka_input leader=false member_id="prodmheventhubs.servicebus.windows.net:c:traffic-streams-prod-mhbe-events-consumer-v3:I:redpanda-connect-19aab9f93a5d4b9facdd45c8ffcd0651" path=root.input.broker.inputs.0
time="2026-07-22T01:51:08Z" level=debug msg=syncing @service=benthos group=traffic-streams-prod-mhbe-events-consumer-v3 label=kafka_input path=root.input.broker.inputs.0 protocol=cooperative-sticky protocol_type=consumer
time="2026-07-22T01:51:08Z" level=debug msg=synced @service=benthos assigned="processedevent[1]" group=traffic-streams-prod-mhbe-events-consumer-v3 label=kafka_input path=root.input.broker.inputs.0
time="2026-07-22T01:51:08Z" level=debug msg="new group session begun" @service=benthos added="" group=traffic-streams-prod-mhbe-events-consumer-v3 label=kafka_input lost="" path=root.input.broker.inputs.0
time="2026-07-22T01:51:08Z" level=debug msg="beginning heartbeat loop" @service=benthos group=traffic-streams-prod-mhbe-events-consumer-v3 label=kafka_input path=root.input.broker.inputs.0
time="2026-07-22T01:51:10Z" level=debug msg="metadata update triggered" @service=benthos label=kafka_input path=root.input.broker.inputs.0 why="opportunistic load during source backoff: read tcp 10.244.189.38:39672->20.86.89.184:9093: i/o timeout"
time="2026-07-22T02:06:48Z" level=debug msg="Successfully dispatched [79] documents in 72ms (1097.222222 docs/sec)" @service=benthos label="" path=root.output.broker.outputs.0
time="2026-07-22T02:06:55Z" level=debug msg="Successfully dispatched [2288] documents in 4.022s (568.871208 docs/sec)" @service=benthos label="" path=root.output.broker.outputs.0
time="2026-07-22T02:06:57Z" level=debug msg="Successfully dispatched [546] documents in 1.267s (430.939227 docs/sec)" @service=benthos label="" path=root.output.broker.outputs.0
time="2026-07-22T02:06:59Z" level=debug msg="Successfully dispatched [804] documents in 1.367s (588.149232 docs/sec)" @service=benthos label="" path=root.output.broker.outputs.0
time="2026-07-22T02:07:05Z" level=debug msg="Successfully dispatched [18] documents in 25ms (720.000000 docs/sec)" @service=benthos label="" path=root.output.broker.outputs.0
time="2026-07-22T02:07:10Z" level=debug msg="Successfully dispatched [7] documents in 12ms (583.333333 docs/sec)" @service=benthos label="" path=root.output.broker.outputs.0
time="2026-07-22T02:07:11Z" level=debug msg="Successfully dispatched [6] documents in 18ms (333.333333 docs/sec)" @service=benthos label="" path=root.output.broker.outputs.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the redpanda and legacy Kafka input entry points, then trace the logged DescribeGroups, Metadata, heartbeat, and commit requests around the rebalance. Reproduce against Event Hubs with the supplied timeout settings and capture client and network timing; done means identifying the cause of the 900-second partition stalls and verifying a fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, go
Domain
data-engineering, networking, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.