apache / apache/iceberg-python

ArrowScan materializes entire FileScanTask into memory

未关闭
#3,036 6 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
stale
主要语言
Python
星标
1.1k
派生
581
平均合并
1 天 17 小时
30 天内合并 PR
78

描述

### Feature Request / Improvement

ArrowScan in PyIceberg does not support true streaming, leading to OOM failures when processing large files (file size > container size). While the API returns an iterator, the implementation eagerly materializes all record batches for a FileScanTask before yielding the first row.

Two primary bottlenecks were identified in the `pyiceberg.io.pyarrow` implementation:
1. The internal scan logic uses a `list()` constructor on the batch iterator, forcing the entire file into memory.
2. The `batch_size` parameter is not forwarded to the underlying PyArrow ds.Scanner, preventing granular memory control. Though, it does fallback to the standard

This behavior makes it impossible to process files larger than the available memory in distributed environments (e.g., Ray workers)

贡献指南

这个仓库没有索引到贡献指南

调研方向

从 pyiceberg.io.pyarrow 开始,跟踪 ArrowScan 的 FileScanTask 批次迭代以及 PyArrow ds.Scanner 的构造。确认批次是在不预先完整物化迭代器的情况下生成的,并且 batch_size 能传递到 scanner;当大文件无需将整个任务放入内存即可处理时,即表示完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
data-engineering, performance
Issue 类型
功能
难度
3/5
预计耗时
1-2 天
活跃度
活跃
描述清晰度
基本清楚
新手友好度
65/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。