apache / apache/iceberg-python
Add maintenance action to remove dangling delete files
- Langage dominant
- Python
- Étoiles
- 1.1k
- Forks
- 581
- Merge moyen
- 1 j 17 h
- PR mergées (30 j)
- 77
Description
### Feature Request / Improvement
Streaming upsert writers (e.g. AWS Firehose Iceberg delivery) write equality-delete files on every commit. Compaction applies them into rewritten data files but leaves the entries in the manifests, and `expire_snapshots` can't touch files the current snapshot still references — so they accumulate without bound. On one of our production tables we measured ~90K dangling delete entries growing ~4.6K/day, and every query planning over recent partitions has to read the ever-growing delete manifests.
Java Iceberg handles this (`rewrite_data_files` with `remove-dangling-deletes`, `RemoveDanglingDeletesSparkAction`), but PyIceberg's `MaintenanceTable` currently only has `expire_snapshots`, and engines like Athena expose no statement for it either — so users on Athena/Firehose stacks have no non-Spark way out.
**Proposal:** `table.maintenance.remove_dangling_deletes()` — a metadata-only commit that:
- classifies per `(partition_spec_id, partition)`: an equality delete at sequence *s* is dangling iff no live data file in that partition has sequence < *s* (position deletes: <= *s*); ambiguous cases (unpartitioned specs, unknown content) are kept
- carries data manifests through unchanged, drops fully-dangling delete manifests, rewrites mixed ones to their surviving entries, and commits as a `replace` snapshot against the current ref
One enabler is worth a small standalone fix first: `ManifestWriterV2` hardcodes `content=data`, so PyIceberg currently can't write delete-content manifests at all.
We have a working implementation built on PyIceberg 0.12 internals (`write_manifest_list`, a `ManifestWriterV2` subclass, `AddSnapshotUpdate`/`SetSnapshotRefUpdate` with `AssertRefSnapshotId`), validated against production Glue/Athena tables, with a test matrix for the classification rules. Happy to contribute it if there's interest.
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Piste de recherche
Commencez par MaintenanceTable et ManifestWriterV2, puis examinez les composants internes référencés de write_manifest_list et de la mise à jour du snapshot. Utilisez les règles de séquence par partition indiquées et la matrice de tests de classification comme critères d’acceptation ; le travail est terminé lorsqu’un commit de remplacement portant uniquement sur les métadonnées supprime seulement les suppressions orphelines tout en préservant les entrées ambiguës et actives.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- aws, python
- Domaine
- data-engineering, databases
- Type d'issue
- Fonctionnalité
- Difficulté
- 5/5
- Temps estimé
- Plus d'une semaine
- Activité
- Active
- Clarté
- Plutôt claire
- Accessibilité débutants
- 45/100