apache / apache/hudi

[SUPPORT] TransactionParticipant should be created for each assigned partition, but has not been created for the topic/partition

Open
#13,028 3 comments 0 reactions 0 assignees View on GitHub
engine:kafka-connect type:community-support
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

I am trying to run the demo with kafka-connect and hudi-sink but when I create the sink connector I get the error TransactionParticipant should be created for each assigned partition. If I recreate the sink after the topic has received data it gives me a different error. I have to delete the sink connector delete the topic and then recreate it to get rid of the connectException error. But I am still stuck with TransactionParticipant error.

**To Reproduce**

Steps to reproduce the behavior:

Follow the instructions given [here](https://github.com/apache/hudi/blob/master/hudi-kafka-connect/README.md)

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment Description**

* Hudi version : 1.0.1

* Spark version : 3.5.5

* Hive version : 2.3.4

* Hadoop version : 2.10.2

* Storage (HDFS/S3/GCS..) : HDFS

* kafka-connect-hdfs: 10.1.0

* Running on Docker? (yes/no) : No

**Additional context**
this is my `connect-distributed.properties`

```bash
sudo cat /etc/kafka/connect-distributed.properties
# Written by Ansible
bootstrap.servers=hudi:9092
config.providers.secret.param.kafkastore.security.protocol=PLAINTEXT
config.storage.replication.factor=1
config.storage.topic=connect-cluster-configs
confluent.license.topic=_confluent-command
confluent.monitoring.interceptor.topic=_confluent-monitoring
connector.client.config.override.policy=All
consumer.bootstrap.servers=hudi:9092
consumer.confluent.monitoring.interceptor.bootstrap.servers=hudi:9092
consumer.confluent.monitoring.interceptor.security.protocol=PLAINTEXT
consumer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor
consumer.security.protocol=PLAINTEXT
group.id=hudi-connect-cluster
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter.schemas.enable=false
key.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
key.converter.schema.registry.url=http://hudi:8081
listeners=http://0.0.0.0:8083
offset.flush.interval.ms=10000
offset.storage.replication.factor=1
offset.storage.topic=connect-cluster-offsets
plugin.path=/usr/share/java/connect_plugins,/usr/share/confluent-hub-components
producer.bootstrap.servers=hudi:9092
producer.confluent.monitoring.interceptor.bootstrap.servers=hudi:9092
producer.confluent.monitoring.interceptor.security.protocol=PLAINTEXT
producer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor
producer.security.protocol=PLAINTEXT
rest.advertised.host.name=hudi
rest.advertised.listener=http
rest.advertised.port=8083
rest.port=8083
security.protocol=PLAINTEXT
status.storage.replication.factor=1
status.storage.topic=connect-cluster-status
value.converter.schema.registry.url=http://hudi:8081
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=true
```

**Stacktrace**

```
.connect.runtime.WorkerSinkTask:617)
Mar 25 05:54:55 hudi connect-distributed[3710459]: org.apache.kafka.connect.errors.RetriableException: TransactionParticipant should be created for each assigned partition, but has not been created for the topic/partition: hudi-test-topic:0
Mar 25 05:54:55 hudi connect-distributed[3710459]: at org.apache.hudi.connect.HoodieSinkTask.put(HoodieSinkTask.java:111)
Mar 25 05:54:55 hudi connect-distributed[3710459]: at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:593)
Mar 25 05:54:55 hudi connect-distributed[3710459]: at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:340)
Mar 25 05:54:55 hudi connect-distributed[3710459]: at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:238)
Mar 25 05:54:55 hudi connect-distributed[3710459]: at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:207)
Mar 25 05:54:55 hudi connect-distributed[3710459]: at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:229)
Mar 25 05:54:55 hudi connect-distributed[3710459]: at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:284)
Mar 25 05:54:55 hudi connect-distributed[3710459]: at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:181)
Mar 25 05:54:55 hudi connect-distributed[3710459]: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
Mar 25 05:54:55 hudi connect-distributed[3710459]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
Mar 25 05:54:55 hudi connect-distributed[3710459]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
Mar 25 05:54:55 hudi connect-distributed[3710459]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
Mar 25 05:54:55 hudi connect-distributed[3710459]: at java.base/java.lang.Thread.run(Thread.java:840)
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the kafka-connect README reproduction steps and inspect org.apache.hudi.connect.HoodieSinkTask.java at line 111, where the stack trace reports the failure. Compare connector assignment and transaction setup for hudi-test-topic:0, then verify that the demo sink starts and processes assigned partitions without the reported exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.