apache / apache/iceberg-python

Delegate reading DataFiles to Iceberg-Rust

未关闭
#2,396 2 条评论 6 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
1.1k
派生
581
平均合并
1 天 17 小时
30 天内合并 PR
78

描述

### Feature Request / Improvement

Today we highly lean on PyArrow to do the reading of the Parquet files, but this has some big disadvantages:

- PyArrow does not treat Field-IDs as first class citizens. Therefore we have to first get the physical schema (from the Parquet files) and [prune the schema](https://github.com/apache/iceberg-python/blob/3eecdadc000047ec30749fc5d6ce1f2f072a30b2/pyiceberg/io/pyarrow.py#L1516) based on field-IDs.
- We have to post-process the buffers to apply schema evolution. For example, if a table has promoted an integer to a long, Iceberg does not rewrite the datafiles with the new column. Instead, when we see an integer at read-time, we [promote the buffer to a long](https://github.com/apache/iceberg-python/blob/3eecdadc000047ec30749fc5d6ce1f2f072a30b2/pyiceberg/io/pyarrow.py#L1554-L1560). Ideally we want to push this down to the reader right away.

If we could push this down into Iceberg-Rust, and return references to Arrow buffers back to PyIceberg, that would be great. We can start simple first by still applying the merge-on-read deletes in PyIceberg, and move that over to Iceberg-Rust step by step.

贡献指南

这个仓库没有索引到贡献指南

调研方向

从 pyiceberg/io/pyarrow.py 开始,重点查看第 1516 行附近的 schema-pruning 代码,以及第 1554-1560 行附近的 buffer promotion。研究 Iceberg-Rust 如何读取 DataFiles 并返回对 Arrow buffers 的引用,同时将 merge-on-read 删除保留在 PyIceberg 中;完成的标准是这项委托的第一步能够正常工作,而无需先移动删除处理逻辑。

由索引模型根据 Issue 内容生成。

评估

技术栈
python, rust
领域
data-engineering
Issue 类型
功能
难度
5/5
预计耗时
一周以上
活跃度
活跃
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。