[SUPPORT] TransactionParticipant not getting created
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
I am trying to add the Hudi Sink Connector to AWS MSK using the below config
connector.class=org.apache.hudi.connect.HoodieSinkConnector
s3.region=us-east-1
hoodie.base.path=s3://data_bucket/
hoodie.table.type=MERGE_ON_READ
flush.size=5
tasks.max=4
topics=******
hoodie.schemaprovider.class=org.apache.hudi.schema.SchemaRegistryProvider
bootstrap.servers=********************
hoodie.datasource.write.partitionpath.field=****
value.converter.region=us-east-1
hoodie.kafka.commit.interval.secs=60
hoodie.table.name=hudi-test
hoodie.datasource.write.recordkey.field=*****
value.converter.schemas.enable=false
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter=org.apache.kafka.connect.storage.StringConverter
The logs show the error message "TransactionParticipant should be created for each assigned partition but has not been created for the topic/partition: " error. I have tried using all versions of Hudi and its the same issue every time.
Under what condition would a TransactionParticipant be not created?
Contributor guide
No contributing guide indexed for this repository
Research direction
The report provides a Hudi Sink Connector configuration and a TransactionParticipant error, but no source file or test. Start by tracing TransactionParticipant creation for assigned topic partitions from the connector entry point, using the supplied configuration and logs; done means the triggering condition and expected behavior are established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, kafka
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100