apache / apache/paimon-rust

Expose DataFrame-style read API (ReadBuilder / Scan / Split / TableRead) to Python

Open
#413 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
197
Forks
92
Avg merge
1d 16h
Merged PRs (30d)
91

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon-rust/issues) and found nothing similar.

### Motivation

PyPaimon has two read paths today:

- **SQL** (`SQLContext.sql`) — already runs on the Rust DataFusion engine.
- **DataFrame** (`ReadBuilder → Split → TableRead.to_arrow/to_pandas/to_ray`) —
still **pure Python**, even though the Rust core already implements the same
model in `crates/paimon/src/table/read_builder.rs`. It's just not exposed
through `bindings/python` (`PyTable` only has `identifier/location/schema`).

Goal: expose the existing Rust read API to Python so the DataFrame read path can
optionally run on Rust. Initially this lands as a **basic, opt-in path behind a
config flag**, running alongside the pure-Python reader rather than replacing it,
so the Rust path can mature before it becomes a default. Write path is out of scope.

## Scope (incremental PRs)

This can be implemented incrementally:

- **PR 1** — Expose scan planning:
`new_read_builder()`, `with_projection()`, `with_limit()`, and
`new_scan().plan()` returning serializable splits.

- **PR 2** — Expose filter pushdown:
add `with_filter()` after the Python Predicate → Rust Predicate conversion
layer is defined.

- **PR 3** — Expose split → Arrow read:
`new_read().read(splits)` returning Arrow data backed by Rust `TableRead`.

- **PR 4** (in `apache/paimon`, `[python]`) — Wire PyPaimon's
`to_arrow` / `to_pandas` / `to_ray` to the Rust reader as an **opt-in path**
(config-gated), keeping the pure-Python reader as the default. Unsupported
capabilities error out rather than silently falling back.

PR 1–3 land here; PR 4 lands in the main repo once bindings are released.

### Notes

`with_filter()` is separated from the initial scan-planning PR because it
requires a dedicated Python Predicate → Rust Predicate conversion layer. PR 1
focuses on establishing the Python binding shape and serializable splits.

Design principle: in this model Rust both **plans and reads**.
`new_read().read(splits)` returns Arrow from the Rust `TableRead`, and splits
stay **opaque** on the Python side — a serializable transport token, not
something Python inspects or reads from. Exposing split internals would imply a
Rust-plans / Python-reads path, which is a different direction and out of scope
here.

### Solution

_No response_

### Anything else?

_No response_

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with crates/paimon/src/table/read_builder.rs and the bindings/python directory, where PyTable currently exposes only identifier, location, and schema. Implement the incremental scan-planning, filter, and Arrow-read bindings described in PRs 1–3, keeping splits opaque and serializable; done means the Rust read API is exposed without changing the Python default reader.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
api, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.