datafusion-contrib / datafusion-contrib/StreamFusion
Propose a Paimon bundle entry on the merge-tree writer with a sequence hand-off
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 120
- Forks
- 8
- Avg merge
- 7h 29m
- Merged PRs (30d)
- 76
Description
Released Paimon 2.0.0 has a bundle write entry for append tables only; its merge-tree writer takes one row at a time and creates level-0 files privately. The native primary-key sink therefore writes level-0 files itself and hands them to Paimon's writer through the entry the dedicated compaction operator uses for files written elsewhere (divergences/32-paimon-pk-l0-through-the-compactor-hook.md). Two costs follow:
- Paimon's writer sees files, not rows, so it can look idle between checkpoints; Paimon then closes it and the next hand-off recreates it with a scan of the bucket's committed files (the same scan its dedicated compactor pays).
- The level-0
DataFileMeta(key bounds, statistics, sequence range, delete count) is assembled on our side from Paimon's public factories rather than by Paimon's own key-value file writer.
Propose upstream a bundle entry on the merge-tree writer that accepts a sorted, key-merged batch for one bucket in Paimon's key-value layout with the caller's sequence range, and creates the level-0 file and its metadata itself. With that entry the compaction hand-off, the idle-writer rescan, and the metadata assembly disappear from StreamFusion; the native sort/merge and Parquet encoding stay. Track the upstream discussion and the Paimon release that ships it here, then switch the sink over and retire divergence 32.
Split from #33.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with divergences/32-paimon-pk-l0-through-the-compactor-hook.md and the context from #33. Track the upstream discussion and the Paimon release that provides the proposed merge-tree bundle entry, then switch the sink over and retire divergence 32 when the hand-off, rescan, and metadata assembly are removed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100