[SUPPORT] Hudi Sink Connector shows broker disconnected
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
I am trying to add the Hudi Sink Connector to AWS MSK using the below config
connector.class=org.apache.hudi.connect.HoodieSinkConnector
s3.region=us-east-1
hoodie.base.path=s3://data_bucket
hoodie.table.type=MERGE_ON_READ
flush.size=5
tasks.max=4
topics=******
hoodie.schemaprovider.class=org.apache.hudi.schema.SchemaRegistryProvider
bootstrap.servers=********************
hoodie.datasource.write.partitionpath.field=****
value.converter.region=us-east-1
hoodie.kafka.commit.interval.secs=60
hoodie.table.name=hudi-test
hoodie.datasource.write.recordkey.field=*****
value.converter.schemas.enable=false
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter=org.apache.kafka.connect.storage.StringConverter
The connector status shows running but theres is no data that getting written to s3. When i check the logs it shows the below error
Bootstrap broker **************************** (id: -2 rack: null) disconnected
I have verified if the brokers are in a healthy state. I have also tested other sink connectors using similar config and those connectors are correctly producing data to s3.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the listed Hudi Sink Connector configuration and the Kafka Connect log entry showing the bootstrap broker disconnect. Reproduce the connector's running-but-no-S3-output behavior on AWS MSK, then establish a concrete cause and verify that records are written successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, kafka
- Domain
- cloud, data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100