datafusion-contrib / datafusion-contrib/StreamFusion
Paimon sink: take the bundle pass-through path once Paimon stops walking bundles row by row
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 120
- Forks
- 8
- Avg merge
- 7h 29m
- Merged PRs (30d)
- 76
Description
Released Paimon 2.0.0 walks a bundle row by row in RowDataFileWriter.writeBundle (counting sequence numbers and, in the buffered spill path, serializing rows) before handing the bundle to the BundleFormatWriter. StreamFusion's bundle rows are zero-copy Arrow views so this costs a per-row cursor move rather than a transpose, but it is still per-row work on the Java side of every batch.
When a Paimon release passes bundles through to the format writer without the row walk (or offers a bundle-level sequence-number update), switch the native writer to that direct path; NativePaimonParquetWriter already encodes the whole batch once and only needs the walk removed. Track the upstream change and re-measure with NexmarkPaimonSinkBenchmark.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Track the Paimon release change affecting RowDataFileWriter.writeBundle and BundleFormatWriter, then inspect NativePaimonParquetWriter for the existing whole-batch encoding path. Once bundle pass-through or bundle-level sequence updates are available, remove the row walk and re-measure with NexmarkPaimonSinkBenchmark.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100