apache / apache/iceberg-rust

PyIceberg-Core: Push down Parquet reading to Iceberg-Rust

Open
#1,144 3 comments 11 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Is your feature request related to a problem or challenge?

As a next step in integrating PyIceberg and Iceberg-Rust, it would be great to push down the Parquet reading (including all the schema evolution) to Iceberg-Rust. Today, in PyIceberg, [we go over each of the record batches](https://github.com/apache/iceberg-python/blob/7a56ddb680313057526a704a6105aea175120c34/pyiceberg/io/pyarrow.py#L1668C6-L1682), which causes a lot of pressure on the GIL. This logic should all happen in the Parquet reader (schema evolution, projecting missing columns, renames, re-ordering, etc), but from PyArrow we don't have the flexibility to project on ID, so this is what we ended up with.

The most logical separation would be to pass the [`FileScanTask` into Iceberg-Rust](https://github.com/apache/iceberg-python/blob/7a56ddb680313057526a704a6105aea175120c34/pyiceberg/table/__init__.py#L1525-L1547).

We can break it down into building blocks:

- Ability to leverage the Iceberg-Rust FileIO in PyIceberg to open up streams
- Ability to pass down a PyIceberg schema into Iceberg-Rust.
- Can we serialize it into JSON? But that seems to be costly. Ideally, we want to reuse objects and not have to copy them from one to the other.
- Pass down expressions.

From the callgraph:

![Image](https://github.com/user-attachments/assets/4d1e864c-49cc-434c-ba79-ef8c283b5d01)

![Image](https://github.com/user-attachments/assets/5ea95b5b-2744-4387-aacf-caff97430882)

### Describe the solution you'd like

_No response_

### Willingness to contribute

None

Contributor guide

Open the contributing guide

Research direction

Start with the PyIceberg call sites in pyiceberg/io/pyarrow.py around lines 1668-1682 and pyiceberg/table/__init__.py around lines 1525-1547, then trace the attached call graph. Identify how FileScanTask, schemas, expressions, and FileIO streams could cross into Iceberg-Rust; done means the Parquet reading and listed schema-evolution operations are handled there.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.