apache / apache/paimon-rust

Expose DataFrame-style read API (ReadBuilder / Scan / Split / TableRead) to Python

オープン
#413 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Rust
スター
197
フォーク
92
平均マージ
1日 16時間
マージ済み PR(30日)
91

説明

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon-rust/issues) and found nothing similar.

### Motivation

PyPaimon has two read paths today:

- **SQL** (`SQLContext.sql`) — already runs on the Rust DataFusion engine.
- **DataFrame** (`ReadBuilder → Split → TableRead.to_arrow/to_pandas/to_ray`) —
still **pure Python**, even though the Rust core already implements the same
model in `crates/paimon/src/table/read_builder.rs`. It's just not exposed
through `bindings/python` (`PyTable` only has `identifier/location/schema`).

Goal: expose the existing Rust read API to Python so the DataFrame read path can
optionally run on Rust. Initially this lands as a **basic, opt-in path behind a
config flag**, running alongside the pure-Python reader rather than replacing it,
so the Rust path can mature before it becomes a default. Write path is out of scope.

## Scope (incremental PRs)

This can be implemented incrementally:

- **PR 1** — Expose scan planning:
`new_read_builder()`, `with_projection()`, `with_limit()`, and
`new_scan().plan()` returning serializable splits.

- **PR 2** — Expose filter pushdown:
add `with_filter()` after the Python Predicate → Rust Predicate conversion
layer is defined.

- **PR 3** — Expose split → Arrow read:
`new_read().read(splits)` returning Arrow data backed by Rust `TableRead`.

- **PR 4** (in `apache/paimon`, `[python]`) — Wire PyPaimon's
`to_arrow` / `to_pandas` / `to_ray` to the Rust reader as an **opt-in path**
(config-gated), keeping the pure-Python reader as the default. Unsupported
capabilities error out rather than silently falling back.

PR 1–3 land here; PR 4 lands in the main repo once bindings are released.

### Notes

`with_filter()` is separated from the initial scan-planning PR because it
requires a dedicated Python Predicate → Rust Predicate conversion layer. PR 1
focuses on establishing the Python binding shape and serializable splits.

Design principle: in this model Rust both **plans and reads**.
`new_read().read(splits)` returns Arrow from the Rust `TableRead`, and splits
stay **opaque** on the Python side — a serializable transport token, not
something Python inspects or reads from. Exposing split internals would imply a
Rust-plans / Python-reads path, which is a different direction and out of scope
here.

### Solution

_No response_

### Anything else?

_No response_

### Willingness to contribute

- [x] I'm willing to submit a PR!

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

crates/paimon/src/table/read_builder.rs と bindings/python ディレクトリから始めてください。そこでは現在、PyTable は identifier、location、schema のみを公開しています。PRs 1–3 で説明されているインクリメンタルなスキャン計画、フィルター、Arrow 読み取りの bindings を実装し、splits は不透明かつシリアライズ可能なままにしてください。完了条件は、Python のデフォルト reader を変更せずに Rust の read API が公開されることです。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
python, rust
領域
api, data-engineering
issue の種類
機能追加
難易度
5/5
見積もり時間
1週間以上
活発さ
静か
明瞭さ
おおむね明確
初心者へのやさしさ
45/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。