[Bug] Paimon CDC not working in Dynamic Bucket Mode
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
The latest code in master branch on Apr 22, commit id = 80040f69d593056e6b8daf18099f87360266e5e7
### Compute Engine
Flink
### Minimal reproduce step
Use `CREATE DATABASE AS` clause in Flink SQL to synchronize an existing database to a Paimon catalog in tests.
### What doesn't meet your expectations?
The following error is thrown in the Flink job
```
Caused by: java.lang.AssertionError
at org.apache.paimon.table.sink.KeyAndBucketExtractor.bucket(KeyAndBucketExtractor.java:53)
at org.apache.paimon.table.sink.FixedBucketRowKeyExtractor.bucket(FixedBucketRowKeyExtractor.java:70)
at org.apache.paimon.flink.sink.RowDataKeyAndBucketExtractor.bucket(RowDataKeyAndBucketExtractor.java:50)
...
```
### Anything else?
I found that the problem is caused by the following invocation process
CdcDynamicBucketSink -> CdcRecordKeyAndBucketExtractor -> KeyAndBucketExtractor.bucket
`CdcDynamicBucketSink` is used when numBuckets = -1, but `KeyAndBucketExtractor.bucket` requires that numBuckets > 0, or an AssertionError would be thrown.
From this point of view, similar problem should also exist for postpone bucket mode, where numBuckets = -2.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the failure with the Flink SQL CREATE DATABASE AS flow, then trace CdcDynamicBucketSink through CdcRecordKeyAndBucketExtractor to KeyAndBucketExtractor.bucket. Review the related FixedBucketRowKeyExtractor and RowDataKeyAndBucketExtractor classes, including the -1 dynamic and -2 postpone bucket cases. Done means the reproduction no longer throws AssertionError for those modes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100