apache / apache/iceberg-python

ArrowScan materializes entire FileScanTask into memory

Offen
#3,036 6 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
stale
Vorherrschende Sprache
Python
Sterne
1.1k
Forks
581
Ø Merge
1 T. 17 Std.
Gemergte PRs (30 T.)
78

Beschreibung

### Feature Request / Improvement

ArrowScan in PyIceberg does not support true streaming, leading to OOM failures when processing large files (file size > container size). While the API returns an iterator, the implementation eagerly materializes all record batches for a FileScanTask before yielding the first row.

Two primary bottlenecks were identified in the `pyiceberg.io.pyarrow` implementation:
1. The internal scan logic uses a `list()` constructor on the batch iterator, forcing the entire file into memory.
2. The `batch_size` parameter is not forwarded to the underlying PyArrow ds.Scanner, preventing granular memory control. Though, it does fallback to the standard

This behavior makes it impossible to process files larger than the available memory in distributed environments (e.g., Ray workers)

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Start in pyiceberg.io.pyarrow by tracing ArrowScan's FileScanTask batch iteration and the construction of the PyArrow ds.Scanner. Confirm that batches are yielded without eagerly materializing the iterator and that batch_size reaches the scanner; done means large files can be processed without requiring the full task in memory.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python
Bereich
data-engineering, performance
Issue-Typ
Feature
Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Aktivitätsstatus
Aktiv
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
65/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.