apache / apache/iceberg-python
Add maintenance action to remove dangling delete files
- Lingua principale
- Python
- Stelle
- 1.1k
- Fork
- 581
- Merge medio
- 1g 17h
- PR unite (30g)
- 77
Descrizione
### Feature Request / Improvement
Streaming upsert writers (e.g. AWS Firehose Iceberg delivery) write equality-delete files on every commit. Compaction applies them into rewritten data files but leaves the entries in the manifests, and `expire_snapshots` can't touch files the current snapshot still references — so they accumulate without bound. On one of our production tables we measured ~90K dangling delete entries growing ~4.6K/day, and every query planning over recent partitions has to read the ever-growing delete manifests.
Java Iceberg handles this (`rewrite_data_files` with `remove-dangling-deletes`, `RemoveDanglingDeletesSparkAction`), but PyIceberg's `MaintenanceTable` currently only has `expire_snapshots`, and engines like Athena expose no statement for it either — so users on Athena/Firehose stacks have no non-Spark way out.
**Proposal:** `table.maintenance.remove_dangling_deletes()` — a metadata-only commit that:
- classifies per `(partition_spec_id, partition)`: an equality delete at sequence *s* is dangling iff no live data file in that partition has sequence < *s* (position deletes: <= *s*); ambiguous cases (unpartitioned specs, unknown content) are kept
- carries data manifests through unchanged, drops fully-dangling delete manifests, rewrites mixed ones to their surviving entries, and commits as a `replace` snapshot against the current ref
One enabler is worth a small standalone fix first: `ManifestWriterV2` hardcodes `content=data`, so PyIceberg currently can't write delete-content manifests at all.
We have a working implementation built on PyIceberg 0.12 internals (`write_manifest_list`, a `ManifestWriterV2` subclass, `AddSnapshotUpdate`/`SetSnapshotRefUpdate` with `AssertRefSnapshotId`), validated against production Glue/Athena tables, with a test matrix for the classification rules. Happy to contribute it if there's interest.
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Direzione di ricerca
Inizia con MaintenanceTable e ManifestWriterV2, quindi esamina gli internals referenziati di write_manifest_list e dell'aggiornamento dello snapshot. Usa le regole di sequenza per partizione indicate e la matrice dei test di classificazione come criteri di accettazione; il lavoro è completo quando un commit di sostituzione che modifica solo i metadati rimuove esclusivamente le eliminazioni orfane, preservando le voci ambigue e attive.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- aws, python
- Ambito
- data-engineering, databases
- Tipo di issue
- Funzionalità
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Stato di attività
- Attiva
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 45/100