apache / apache/paimon

[Feature] Support to run the compact job and the write only job for the primary key table

Open
#5,855 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

For the unaware bucket table, it use the Compact coordinator source to run the write only writer and the compact task in one job. For the primary key table it may also useful.

If the table has the streaming job and batch job write parallelism. we could let the streaming job do the compaction. So the streaming job has to run a compaction source to monitor the table's new created files.

The main benefit is that we could save one extra compaction job.

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no files or tests. Start with the Compact coordinator source used for unaware bucket tables, then trace how primary key tables, streaming jobs, batch write parallelism, and newly created files interact. Done means a primary-key streaming job can monitor new files and run compaction without a separate compaction job.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.