datafusion-contrib / datafusion-contrib/StreamFusion

Propose a Paimon bundle entry on the merge-tree writer with a sequence hand-off

Open
#49 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
120
Forks
8
Avg merge
7h 29m
Merged PRs (30d)
76

Description

Released Paimon 2.0.0 has a bundle write entry for append tables only; its merge-tree writer takes one row at a time and creates level-0 files privately. The native primary-key sink therefore writes level-0 files itself and hands them to Paimon's writer through the entry the dedicated compaction operator uses for files written elsewhere (divergences/32-paimon-pk-l0-through-the-compactor-hook.md). Two costs follow:

  • Paimon's writer sees files, not rows, so it can look idle between checkpoints; Paimon then closes it and the next hand-off recreates it with a scan of the bucket's committed files (the same scan its dedicated compactor pays).
  • The level-0 DataFileMeta (key bounds, statistics, sequence range, delete count) is assembled on our side from Paimon's public factories rather than by Paimon's own key-value file writer.

Propose upstream a bundle entry on the merge-tree writer that accepts a sorted, key-merged batch for one bucket in Paimon's key-value layout with the caller's sequence range, and creates the level-0 file and its metadata itself. With that entry the compaction hand-off, the idle-writer rescan, and the metadata assembly disappear from StreamFusion; the native sort/merge and Parquet encoding stay. Track the upstream discussion and the Paimon release that ships it here, then switch the sink over and retire divergence 32.

Split from #33.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with divergences/32-paimon-pk-l0-through-the-compactor-hook.md and the context from #33. Track the upstream discussion and the Paimon release that provides the proposed merge-tree bundle entry, then switch the sink over and retire divergence 32 when the hand-off, rescan, and metadata assembly are removed.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.