[Bug] Combine mode sink only supports FIXED bucket mode, but table is DYNAMIC
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
0.8.0
### Compute Engine
flink
### Minimal reproduce step
sync database from kafka topic, when the table bucket is -1,the error:java.lang.UnsupportedoperationException: Combine mode Sink only supports FIxED bucket mode, but org t main is DYNAMIC;
my command is:
flink run-application -t yarn-application \
-Dyarn.application.name="paimon-load-realtime-orc-test" \
-Dpipeline.name='paimon-canal-databasesync-orc-test' \
-Djobmanager.memory.process.size=4096m \
-Dtaskmanager.memory.process.size=4096m \
-Dexecution.checkpointing.interval=60000 \
-Dexecution.checkpointing.timeout=60000 \
-Dexecution.checkpointing.tolerable-failed-checkpoints=3 \
-Dstate.backend.type='rocksdb' \
-Dexecution.checkpointing.checkpoints-after-tasks-finish.enabled='true' \
-Dpipeline.operator-chaining='false' \
-Dstate.savepoints.dir='hdfs://mycluster/flink/savepoints' \
-Dstate.checkpoints.dir='hdfs://mycluster/flink/checkpoints' \
-Dstate.checkpoints.num-retained=100 \
-Dcluster.evenly-spread-out-slots='true' \
/opt/bigdata/flink/lib/paimon-flink-action-0.8.0.jar \
kafka-sync-database \
--warehouse hdfs://mycluster/user/paimon/warehouse \
--database paimon_odata \
--table-prefix "" \
--table-suffix "" \
--kafka-conf properties.bootstrap.servers='node1:9092,node2:9092,node3:9092' \
--kafka-conf topic="new_tables_canal" \
--kafka-conf properties.security.protocol=SASL_PLAINTEXT \
--kafka-conf properties.sasl.mechanism=PLAIN \
--kafka-conf properties.sasl.jaas.config='org.apache.kafka.common.security.plain.PlainLoginModule required username="client" password="";' \
--kafka-conf properties.group.id=paimon-canal-orc-test \
--kafka-conf scan.startup.mode=earliest-offset \
--kafka-conf value.format=canal-json \
--catalog-conf metastore=hive \
--catalog-conf uri=thrift://node2:9083 \
--table-conf bucket=-1 \
--table-conf changelog-producer=lookup \
--table-conf sink.parallelism=32 \
--including_tables org_t_main
### What doesn't meet your expectations?
cant sync table or database,the error:

### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the provided Flink kafka-sync-database command and reproduce the failure using bucket=-1 with changelog-producer=lookup. Inspect the combine-mode sink path to understand its handling of dynamic buckets, then add a regression test covering this configuration. Done means the database or table sync no longer fails unexpectedly, or the supported limitation is made explicit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100