PyIceberg-Core: Push down Parquet reading to Iceberg-Rust
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Is your feature request related to a problem or challenge?
As a next step in integrating PyIceberg and Iceberg-Rust, it would be great to push down the Parquet reading (including all the schema evolution) to Iceberg-Rust. Today, in PyIceberg, [we go over each of the record batches](https://github.com/apache/iceberg-python/blob/7a56ddb680313057526a704a6105aea175120c34/pyiceberg/io/pyarrow.py#L1668C6-L1682), which causes a lot of pressure on the GIL. This logic should all happen in the Parquet reader (schema evolution, projecting missing columns, renames, re-ordering, etc), but from PyArrow we don't have the flexibility to project on ID, so this is what we ended up with.
The most logical separation would be to pass the [`FileScanTask` into Iceberg-Rust](https://github.com/apache/iceberg-python/blob/7a56ddb680313057526a704a6105aea175120c34/pyiceberg/table/__init__.py#L1525-L1547).
We can break it down into building blocks:
- Ability to leverage the Iceberg-Rust FileIO in PyIceberg to open up streams
- Ability to pass down a PyIceberg schema into Iceberg-Rust.
- Can we serialize it into JSON? But that seems to be costly. Ideally, we want to reuse objects and not have to copy them from one to the other.
- Pass down expressions.
From the callgraph:


### Describe the solution you'd like
_No response_
### Willingness to contribute
None
Contributor guide
Research direction
Start with the PyIceberg call sites in pyiceberg/io/pyarrow.py around lines 1668-1682 and pyiceberg/table/__init__.py around lines 1525-1547, then trace the attached call graph. Identify how FileScanTask, schemas, expressions, and FileIO streams could cross into Iceberg-Rust; done means the Parquet reading and listed schema-evolution operations are handled there.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100