datafusion-contrib / datafusion-contrib/StreamFusion
Paimon streaming source: remaining native scan coverage
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 120
- Forks
- 8
- Avg merge
- 7h 29m
- Merged PRs (30d)
- 76
Description
Remaining coverage for the streaming Paimon source. Java owns table discovery, split assignment, FileIO and checkpoint formats; admitted Parquet and ORC append data and produced primary-key changelogs decode directly into Arrow. ORC coverage ships in 446d39c0. e2d2fc60 adds native initial snapshot merging for admitted deduplication splits, raw snapshot decoding, and Paimon-compatible explicit source UIDs for savepoint restoration. Other snapshot combinations retain Java. Connector coverage is the source of truth for current behavior and exact admission gates.
Compatible repeated scans now share native Parquet/ORC readers across projections and statement-set sinks (d1c7c2b6); the unchanged upstream source-reuse assertion passes.
Work still to do:
- Broader native initial snapshot merging, tracked separately in #53.
- Projection and source-ability coverage beyond top-level fields: nested subfield pruning, zero-column scans, metadata columns and applicable pushed limits/aggregates.
- Safe predicate pruning/pushdown. Query predicates currently remain Flink residuals; the source does not push query filters into its Java scan or native decoder.
- Native decoding for historical schema mapping and deletion-vector selections. Those splits currently use Java reading and convert the logical rows to Arrow.
- Broader formats and value types beyond the admitted Parquet/ORC schema contracts, including the documented ORC timestamp/timezone restrictions.
- Verify and admit additional streaming source configurations: consumer retention, checkpoint/snapshot alignment, dedicated split generation, advanced startup/read options, internal row/sequence metadata and watermark modes outside the shared periodic contract. Each addition needs explicit stock parity and recovery coverage before removing its fallback.
- Evaluate streaming primary-key reads without a changelog producer as a separate admission extension; the current implementation requires produced changelog files.
All work here is streaming only. Batch reads and batch clustering remain with stock Paimon. Sink gaps remain in their individual issues and the connector coverage page.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with docs/connectors/paimon.md#streaming-source, which defines current behavior and admission gates, and review the existing native snapshot and Parquet/ORC coverage referenced in the issue. Use the unchanged upstream source-reuse assertion and the listed recovery requirements to verify each configuration or decoding extension before removing its Java fallback.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100