Paimon streaming source: remaining native snapshot merge coverage

Preferred direction for the streaming Paimon source: Java Paimon plans the splits and owns enumeration/checkpointing and all FileIO; paimon-rust reads those supplied splits, decodes files and performs snapshot merging into Arrow. Prefer the public Rust reader as a dependency over continuing to maintain copied reader/merge implementations, subject to released-API availability and demonstrated Java parity. This is the target architecture; the implementation currently shipped is described below.

Ownership boundary

• Java Paimon: catalog/table and schema resolution, snapshots/manifests, split planning and assignment, discovery/enumeration, checkpoint serialization and recovery, and the actual FileIO implementation/authentication.
• paimon-rust: consume the supplied schema and split/file descriptors through its public reader API, decode Parquet/ORC and apply supported read/merge logic, returning Arrow batches.
• Java-backed storage adapter: every native request for file bytes or filesystem metadata goes through the existing Java Paimon FileIO. Preserve remote URI and external-file-path semantics, credentials, filesystem plugins, stream cleanup, cancellation and storage exceptions. Native code must not instantiate a second object-store/HDFS client or independently discover newer snapshots.
• StreamFusion: bridge descriptors and byte requests over JNI, import Arrow output, carry row kinds, and checkpoint emitted logical rows after merging/delete removal. Java/native restoration must replay the same logical prefix when batch sizes change. Keep decoded and merged data in Arrow without a Java-row round trip.

Implementation and admission

1. Confirm the canonical released crate exposes a usable caller-supplied FileIO backend. The checkout has public Table::from_resolved_schema, DataSplit, and TableRead::to_arrow(&splits), so the public reader can consume Java-planned inputs without using Rust enumeration. The individual merge helpers remain private.
2. Build the Java FileIO adapter and descriptor translation, reusing our existing byte-read and Arrow ownership patterns. Prove remote paths as well as local files, concurrent reads, exceptions, close/cancel, and that every storage operation reaches Java FileIO.
3. Start with a small explicit whitelist proven against released Java Paimon, then expand using the existing SQL reader, type, projection, delete/sequence, snapshot-to-changelog and Java/native recovery fixtures. Do not inherit the current native whitelist merely because the Rust API accepts an option. Choose compatibility fallback before emission; fail/recover on runtime storage/resource errors.
4. Benchmark Java-to-Arrow, the current native path, and the paimon-rust-backed path with release builds. Remove duplicated source reader/merge code only as the replacement passes parity and performance checks. Batch SQL remains stock Paimon.

Released-API finding: the crates.io index currently lists paimon 0.3.0 as the latest release. Inspection of that published crate confirms public TableRead::to_arrow and Table::new, but no public custom FileIO-provider/operator injection. The local 0.4.0 checkout adds with_fs_operator, whose documented contract accepts filesystem paths and rejects object-store URIs; the general FileIOProvider/with_provider API remains crate-private. The public FileRead trait alone cannot be installed in the full reader. Resolve this dependency/API prerequisite before claiming a pure-import implementation can preserve arbitrary Java FileIO. Do not silently switch storage ownership or add an unreleased dependency.

Rough engineering estimate: 1–2 working days for a narrow working prototype and 3–5 working days total for an initial shippable whitelist with both-format parity/recovery/cleanup tests, release benchmarks and CI, assuming a suitable released FileIO hook. Upstream API/release waiting time is separate. This estimate covers the adapter and first proven coverage, not all remaining combinations below.

Current implementation

Connector coverage is authoritative for admission, memory limits, tests and release benchmarks. The latest increment (3c3d020e) adds basic partial-update merging by selecting cells in Arrow, including ignore-delete and whole-record delete/reinsert behavior. A port of Java's loser-tree states also admits exact stored-sequence ties and overlapping file sequence intervals. Fixed/dynamic hash buckets, scalar user sequences and projections retain Java-compatible logical-row restoration. No Java row conversion or one-row Arrow allocation occurs between file decoding and merging.

Validation includes 653 Paimon regression cases, 378 direct Java-oracle combinations, 78 Parquet/ORC tie/value cases, streaming SQL and Java/native restores, and eight unchanged upstream SQL tests with native execution markers. Release partial-update catch-up measured 1.78–1.82× Java throughput with Parquet and 1.79–1.89× with ORC; the ORC fixture uses the documented 256 MiB admission budget. Full methodology and limits are in the connector page.

Coverage to whitelist incrementally:

• Extend partial updates to sequence groups, removal by sequence group and configured field aggregates; add aggregation-engine support. Reuse relevant paimon-rust or existing writer components where useful. Verify postpone-bucket and cross-partition key-dynamic snapshot combinations separately.
• Extend timestamp precision above 6 while preserving Paimon's ordering and Arrow representation.
• Broaden ORC timestamp admission: non-UTC timestamp schemas, non-leading fractional timestamp stored keys, and leading key bounds intersecting the last-negative-second alias currently retain Java.
• Integrate schema evolution and deletion-vector selection with native snapshot merging, coordinated with #27.
• Extend memory coverage beyond fan-in/footer admission and the retained-Arrow budget: native spilling or another proven path for larger sections, and accounting/reservations for decoder, output and handover memory. Current limits are not a strict total-process or Flink managed-memory reservation.
• For each extension, verify exact stock/native rows and kinds, projections, cancellation/failure cleanup, mid-split restoration in both Java/native directions, and SQL snapshot-to-changelog recovery. Benchmark catch-up with release libraries.

Unsupported combinations retain the Java snapshot reader. Batch SQL remains with stock Paimon. Use canonical released dependencies. Prefer importing the public paimon-rust reader once its FileIO boundary can preserve the ownership contract above; use focused attributed ports only where an API or Java-compatibility gap makes them necessary. Do not add fork/Git/path/snapshot dependencies.

Follow-up to #27. The design records the planning and ownership boundary.

Start with docs/connectors/paimon.md and .claude/research/paimon-native-snapshot-merge-design.md, then verify the released paimon 0.3.0 API and its FileIO limitations before choosing an implementation path. Coordinate schema evolution and deletion-vector work with #27. Done requires a released-compatible storage boundary, Java/native row and recovery parity, cleanup and failure tests, and release benchmarks for the admitted coverage.