apache / apache/iceberg-python

Pluggable Backend Interface with DataFusion for Bounded-Memory Compute

未關閉
#3,715 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Python
星號
1.1k
分支
588
平均合併
1 天 23 小時
30 天內合併 PR
84

描述

## Summary

PyIceberg uses PyArrow as its sole execution engine. PyArrow is a kernel library with no memory management, no spill-to-disk, and no join operators. Operations that process more data than available memory (CoW deletes, equality delete resolution, scan planning for heavily-deleted tables, sorted writes) crash with OOM errors.

This issue tracks introducing a pluggable backend interface (`ReadBackend`, `WriteBackend`, `ComputeBackend` protocols) and integrating Apache DataFusion as the first bounded-memory compute backend.

## Problem

| Operation | Current Status | OOM Pattern |
|-----------|---------------|-------------|
| Equality delete reads | Hard `ValueError` | Anti-join requires all delete keys in memory |
| CoW delete (large files) | OOMs | Materializes entire Parquet file into RAM |
| Scan planning (>100K deletes) | OOMs | All delete entries in Python dict |
| Sort-on-write | Not implemented | Full sort before write |
| Positional deletes (millions) | OOMs | Python set of positions |

Tables written by Flink (which uses equality deletes) are completely unreadable by PyIceberg today.

## Solution

1. **Pluggable interface**: `ReadBackend`, `WriteBackend`, `ComputeBackend` protocols that decouple PyIceberg from PyArrow
2. **DataFusion integration**: Bounded-memory sort, join, and filter with spill-to-disk via `datafusion-python`
3. **Migration**: All existing data operations route through the interface with zero API changes

## Deliverables

- [ ] Equality delete resolution (NEW): tables with equality deletes can now be read
- [ ] CoW delete/overwrite streaming (FIX): statistics short-circuit + two-pass streaming
- [ ] Positional delete resolution (IMPROVED): bounded-memory for large delete sets
- [ ] Sort-on-write (NEW): external merge sort when DataFusion installed
- [ ] Bounded-memory scan planning (NEW): for tables with >100K delete files

## Related Issues

- #1210 - Support reading equality delete files
- #3270 - Equality Delete support
- #3554 - Integrate DataFusion as execution engine

## Acceptance Criteria

- All existing tests pass without `datafusion` installed (no regression)
- Tables with equality deletes return correct results
- CoW delete on 2GB+ files completes without OOM (with DataFusion)
- Sort-on-write produces sorted files when table has sort order and DataFusion installed
- Property-based tests verify PyArrow and DataFusion backends produce identical output

貢獻指南

這個儲存庫沒有索引到貢獻指南

研究方向

未指定實作檔案或測試。先閱讀相關 issue #1210、#3270 和 #3554,然後界定所提議的 ReadBackend、WriteBackend 和 ComputeBackend 協定以及 DataFusion 整合的範圍。完成條件是實作列出的刪除、串流、排序和掃描規劃行為,在沒有 DataFusion 時不發生回歸,且 PyArrow 和 DataFusion 的結果等價。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
python
領域
backend, data-engineering, databases
Issue 類型
功能
難度
5/5
預估耗時
一週以上
活躍度
冷清
描述清晰度
需要釐清
新手友好度
25/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。