apache / apache/paimon

[Feature] Support compaction for CDC Sink for bucket-unaware table

Open
#5,982 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

I noticed compaction is disabled for CDC Sink for unaware bucket tables, as noted in:

https://github.com/apache/paimon/blob/2d825d5e4eafd2e34b5a7300503a1ae79a60dac0/paimon-flink/paimon-flink-cdc/src/main/java/org/apache/paimon/flink/sink/cdc/CdcAppendTableSink.java#L33-L37

We have an unaware bucket append-only table (Kafka to Paimon sink, by extending [KafkaSyncTableAction](https://github.com/apache/paimon/blob/866dd0bd039e4ad3a8da3725f5b01e86f9b64078/paimon-flink/paimon-flink-cdc/src/main/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSyncTableAction.java#L27)), and I'd like to enable compaction to avoid having too many small files. I was wondering if there are any plans to support compaction for this use-case.

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading CdcAppendTableSink.java at the referenced compaction-disabled code and then KafkaSyncTableAction.java to understand the bucket-unaware CDC sink path. Trace how the append-only table is configured and determine what would be required for compaction without buckets. Done means the described CDC Sink use case supports compaction and avoids an excess of small files.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.