[Feature] Support to run the compact job and the write only job for the primary key table
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
For the unaware bucket table, it use the Compact coordinator source to run the write only writer and the compact task in one job. For the primary key table it may also useful.
If the table has the streaming job and batch job write parallelism. we could let the streaming job do the compaction. So the streaming job has to run a compaction source to monitor the table's new created files.
The main benefit is that we could save one extra compaction job.
### Solution
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no files or tests. Start with the Compact coordinator source used for unaware bucket tables, then trace how primary key tables, streaming jobs, batch write parallelism, and newly created files interact. Done means a primary-key streaming job can monitor new files and run compaction without a separate compaction job.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100