apache / apache/hudi

[SUPPORT] TransactionParticipant not getting created

Open
#9,384 12 comments 0 reactions 0 assignees View on GitHub
engine:kafka-connect priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

I am trying to add the Hudi Sink Connector to AWS MSK using the below config

connector.class=org.apache.hudi.connect.HoodieSinkConnector
s3.region=us-east-1
hoodie.base.path=s3://data_bucket/
hoodie.table.type=MERGE_ON_READ
flush.size=5
tasks.max=4
topics=******
hoodie.schemaprovider.class=org.apache.hudi.schema.SchemaRegistryProvider
bootstrap.servers=********************
hoodie.datasource.write.partitionpath.field=****
value.converter.region=us-east-1
hoodie.kafka.commit.interval.secs=60
hoodie.table.name=hudi-test
hoodie.datasource.write.recordkey.field=*****
value.converter.schemas.enable=false
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter=org.apache.kafka.connect.storage.StringConverter

The logs show the error message "TransactionParticipant should be created for each assigned partition but has not been created for the topic/partition: " error. I have tried using all versions of Hudi and its the same issue every time.

Under what condition would a TransactionParticipant be not created?

Contributor guide

No contributing guide indexed for this repository

Research direction

The report provides a Hudi Sink Connector configuration and a TransactionParticipant error, but no source file or test. Start by tracing TransactionParticipant creation for assigned topic partitions from the connector entry point, using the supplied configuration and logs; done means the triggering condition and expected behavior are established.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, kafka
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.