apache / apache/paimon

[Bug] Paimon CDC not working in Dynamic Bucket Mode

Open
#5,534 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

The latest code in master branch on Apr 22, commit id = 80040f69d593056e6b8daf18099f87360266e5e7

### Compute Engine

Flink

### Minimal reproduce step

Use `CREATE DATABASE AS` clause in Flink SQL to synchronize an existing database to a Paimon catalog in tests.

### What doesn't meet your expectations?

The following error is thrown in the Flink job

```
Caused by: java.lang.AssertionError
at org.apache.paimon.table.sink.KeyAndBucketExtractor.bucket(KeyAndBucketExtractor.java:53)
at org.apache.paimon.table.sink.FixedBucketRowKeyExtractor.bucket(FixedBucketRowKeyExtractor.java:70)
at org.apache.paimon.flink.sink.RowDataKeyAndBucketExtractor.bucket(RowDataKeyAndBucketExtractor.java:50)
...
```

### Anything else?

I found that the problem is caused by the following invocation process

CdcDynamicBucketSink -> CdcRecordKeyAndBucketExtractor -> KeyAndBucketExtractor.bucket

`CdcDynamicBucketSink` is used when numBuckets = -1, but `KeyAndBucketExtractor.bucket` requires that numBuckets > 0, or an AssertionError would be thrown.

From this point of view, similar problem should also exist for postpone bucket mode, where numBuckets = -2.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure with the Flink SQL CREATE DATABASE AS flow, then trace CdcDynamicBucketSink through CdcRecordKeyAndBucketExtractor to KeyAndBucketExtractor.bucket. Review the related FixedBucketRowKeyExtractor and RowDataKeyAndBucketExtractor classes, including the -1 dynamic and -2 postpone bucket cases. Done means the reproduction no longer throws AssertionError for those modes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.