apache / apache/paimon

[Bug] Combine mode sink only supports FIXED bucket mode, but table is DYNAMIC

Open
#3,321 0 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

0.8.0

### Compute Engine

flink

### Minimal reproduce step

sync database from kafka topic, when the table bucket is -1,the error:java.lang.UnsupportedoperationException: Combine mode Sink only supports FIxED bucket mode, but org t main is DYNAMIC;

my command is:
flink run-application -t yarn-application \
-Dyarn.application.name="paimon-load-realtime-orc-test" \
-Dpipeline.name='paimon-canal-databasesync-orc-test' \
-Djobmanager.memory.process.size=4096m \
-Dtaskmanager.memory.process.size=4096m \
-Dexecution.checkpointing.interval=60000 \
-Dexecution.checkpointing.timeout=60000 \
-Dexecution.checkpointing.tolerable-failed-checkpoints=3 \
-Dstate.backend.type='rocksdb' \
-Dexecution.checkpointing.checkpoints-after-tasks-finish.enabled='true' \
-Dpipeline.operator-chaining='false' \
-Dstate.savepoints.dir='hdfs://mycluster/flink/savepoints' \
-Dstate.checkpoints.dir='hdfs://mycluster/flink/checkpoints' \
-Dstate.checkpoints.num-retained=100 \
-Dcluster.evenly-spread-out-slots='true' \
/opt/bigdata/flink/lib/paimon-flink-action-0.8.0.jar \
kafka-sync-database \
--warehouse hdfs://mycluster/user/paimon/warehouse \
--database paimon_odata \
--table-prefix "" \
--table-suffix "" \
--kafka-conf properties.bootstrap.servers='node1:9092,node2:9092,node3:9092' \
--kafka-conf topic="new_tables_canal" \
--kafka-conf properties.security.protocol=SASL_PLAINTEXT \
--kafka-conf properties.sasl.mechanism=PLAIN \
--kafka-conf properties.sasl.jaas.config='org.apache.kafka.common.security.plain.PlainLoginModule required username="client" password="";' \
--kafka-conf properties.group.id=paimon-canal-orc-test \
--kafka-conf scan.startup.mode=earliest-offset \
--kafka-conf value.format=canal-json \
--catalog-conf metastore=hive \
--catalog-conf uri=thrift://node2:9083 \
--table-conf bucket=-1 \
--table-conf changelog-producer=lookup \
--table-conf sink.parallelism=32 \
--including_tables org_t_main

### What doesn't meet your expectations?

cant sync table or database,the error:
![image](https://github.com/apache/paimon/assets/20592169/123868db-e942-43cb-8d54-c1a1bea2691f)

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the provided Flink kafka-sync-database command and reproduce the failure using bucket=-1 with changelog-producer=lookup. Inspect the combine-mode sink path to understand its handling of dynamic buckets, then add a regression test covering this configuration. Done means the database or table sync no longer fails unexpectedly, or the supported limitation is made explicit.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.