apache / apache/iceberg-python

Pluggable Backend Interface with DataFusion for Bounded-Memory Compute

Ouverte
#3,715 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Python
Étoiles
1.1k
Forks
581
Merge moyen
1 j 17 h
PR mergées (30 j)
78

Description

## Summary

PyIceberg uses PyArrow as its sole execution engine. PyArrow is a kernel library with no memory management, no spill-to-disk, and no join operators. Operations that process more data than available memory (CoW deletes, equality delete resolution, scan planning for heavily-deleted tables, sorted writes) crash with OOM errors.

This issue tracks introducing a pluggable backend interface (`ReadBackend`, `WriteBackend`, `ComputeBackend` protocols) and integrating Apache DataFusion as the first bounded-memory compute backend.

## Problem

| Operation | Current Status | OOM Pattern |
|-----------|---------------|-------------|
| Equality delete reads | Hard `ValueError` | Anti-join requires all delete keys in memory |
| CoW delete (large files) | OOMs | Materializes entire Parquet file into RAM |
| Scan planning (>100K deletes) | OOMs | All delete entries in Python dict |
| Sort-on-write | Not implemented | Full sort before write |
| Positional deletes (millions) | OOMs | Python set of positions |

Tables written by Flink (which uses equality deletes) are completely unreadable by PyIceberg today.

## Solution

1. **Pluggable interface**: `ReadBackend`, `WriteBackend`, `ComputeBackend` protocols that decouple PyIceberg from PyArrow
2. **DataFusion integration**: Bounded-memory sort, join, and filter with spill-to-disk via `datafusion-python`
3. **Migration**: All existing data operations route through the interface with zero API changes

## Deliverables

- [ ] Equality delete resolution (NEW): tables with equality deletes can now be read
- [ ] CoW delete/overwrite streaming (FIX): statistics short-circuit + two-pass streaming
- [ ] Positional delete resolution (IMPROVED): bounded-memory for large delete sets
- [ ] Sort-on-write (NEW): external merge sort when DataFusion installed
- [ ] Bounded-memory scan planning (NEW): for tables with >100K delete files

## Related Issues

- #1210 - Support reading equality delete files
- #3270 - Equality Delete support
- #3554 - Integrate DataFusion as execution engine

## Acceptance Criteria

- All existing tests pass without `datafusion` installed (no regression)
- Tables with equality deletes return correct results
- CoW delete on 2GB+ files completes without OOM (with DataFusion)
- Sort-on-write produces sorted files when table has sort order and DataFusion installed
- Property-based tests verify PyArrow and DataFusion backends produce identical output

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Piste de recherche

Aucun fichier d’implémentation ni test n’est nommé. Commencez par lire les issues associées #1210, #3270 et #3554, puis définissez le périmètre des protocoles ReadBackend, WriteBackend et ComputeBackend proposés ainsi que de l’intégration DataFusion. Le travail est considéré comme terminé lorsque les comportements listés de suppression, de streaming, de tri et de planification des scans sont pris en charge, qu’il n’y a aucune régression sans DataFusion et que les résultats de PyArrow et de DataFusion sont équivalents.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
python
Domaine
backend, data-engineering, databases
Type d'issue
Fonctionnalité
Difficulté
5/5
Temps estimé
Plus d'une semaine
Activité
Calme
Clarté
À clarifier
Accessibilité débutants
25/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.