apache / apache/iceberg-python

ArrowScan materializes entire FileScanTask into memory

Open
#3,036 6 comments 0 reactions 0 assignees View on GitHub
stale
Dominant language
Python
Stars
1.1k
Forks
581
Avg merge
1d 17h
Merged PRs (30d)
78

Description

### Feature Request / Improvement

ArrowScan in PyIceberg does not support true streaming, leading to OOM failures when processing large files (file size > container size). While the API returns an iterator, the implementation eagerly materializes all record batches for a FileScanTask before yielding the first row.

Two primary bottlenecks were identified in the `pyiceberg.io.pyarrow` implementation:
1. The internal scan logic uses a `list()` constructor on the batch iterator, forcing the entire file into memory.
2. The `batch_size` parameter is not forwarded to the underlying PyArrow ds.Scanner, preventing granular memory control. Though, it does fallback to the standard

This behavior makes it impossible to process files larger than the available memory in distributed environments (e.g., Ray workers)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.