apache / apache/hudi

[SUPPORT] Hudi Sink Connector shows broker disconnected

Open
#9,070 11 comments 1 reaction 0 assignees View on GitHub
engine:kafka-connect priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

I am trying to add the Hudi Sink Connector to AWS MSK using the below config

connector.class=org.apache.hudi.connect.HoodieSinkConnector
s3.region=us-east-1
hoodie.base.path=s3://data_bucket
hoodie.table.type=MERGE_ON_READ
flush.size=5
tasks.max=4
topics=******
hoodie.schemaprovider.class=org.apache.hudi.schema.SchemaRegistryProvider
bootstrap.servers=********************
hoodie.datasource.write.partitionpath.field=****
value.converter.region=us-east-1
hoodie.kafka.commit.interval.secs=60
hoodie.table.name=hudi-test
hoodie.datasource.write.recordkey.field=*****
value.converter.schemas.enable=false
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter=org.apache.kafka.connect.storage.StringConverter

The connector status shows running but theres is no data that getting written to s3. When i check the logs it shows the below error

Bootstrap broker **************************** (id: -2 rack: null) disconnected

I have verified if the brokers are in a healthy state. I have also tested other sink connectors using similar config and those connectors are correctly producing data to s3.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the listed Hudi Sink Connector configuration and the Kafka Connect log entry showing the bootstrap broker disconnect. Reproduce the connector's running-but-no-S3-output behavior on AWS MSK, then establish a concrete cause and verify that records are written successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, kafka
Domain
cloud, data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.