apache / apache/iceberg-python

Support merge-on-read deletes on v2 tables

Offen
#3,618 1 Kommentar 1 Reaktion 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
1.1k
Forks
581
Ø Merge
1 T. 17 Std.
Gemergte PRs (30 T.)
77

Beschreibung

### Feature Request / Improvement

`Transaction.delete()` is copy-on-write: deleting rows rewrites every data file that holds a matched row, so it costs ~O(table) per operation and grows with the table. Add **v2 merge-on-read position-delete writes** so a delete instead writes a small position-delete file, costing O(rows deleted), with the data files left untouched. v2 position deletes are readable by the existing query engines today, so this is broadly adoptable.

Even as v3 merge-on-read (deletion vectors) support lands, v2 position-delete writes remain valuable, because format version is a per-table property and the two encodings are mutually exclusive: a v2 table can never use a deletion vector. v3 write support therefore does nothing for the large existing v2 fleet, whose tables stay v2 until deliberately upgraded and often must stay v2 to remain readable by every engine that consumes them (v3 deletion vectors require newer readers). For those tables, position deletes are the only efficient delete path, and they are also the format Java/Spark Iceberg already writes, so this also closes a read/write asymmetry in PyIceberg. Deletion vectors are the right default for new v3 tables; v2 position deletes serve the installed base.

#### What already exists (no work needed)

- The read path already **applies** position deletes at scan time (the delete-file index / position-delete application).
- `POSITIONAL_DELETE_SCHEMA` (`file_path`, `pos`), `DataFileContent.POSITION_DELETES`, `ManifestContent.DELETES`, and the full snapshot / manifest-list / commit machinery.
- `Transaction.delete()` already recognizes `write.delete.mode=merge-on-read`. It currently just warns "not yet supported" and falls back to copy-on-write. That fallback is the hook point.

#### The gap to close

1. Widen `pos` in `POSITIONAL_DELETE_SCHEMA` from `int` to `long`. The Iceberg spec types `pos` as `long`; keeping it `int` mismatches what other engines write/read and overflows on files with more than 2^31 rows. The read path treats `pos` as a generic index, so widening is safe.
2. A **DELETES manifest writer**. The v2 manifest writer hardcodes manifest content to `data`; a variant is needed that writes content `deletes`.
3. A **snapshot producer** that adds position-delete files (writes a DELETES manifest for them, grouped by partition spec, and carries existing manifests forward), committing an overwrite snapshot.
4. A **position-delete file writer** that records **exact** `file_path` lower/upper bounds. The read-side delete-file index pins a delete to a single data file only when its `file_path` bound is exact; the default `truncate(16)` write-metrics mode would truncate that bound and mis-route (or over-apply) the delete.
5. **Predicate to positions resolution**. For each data file a delete predicate touches, read it in physical order and collect the ordinal row positions that match, then write one position-delete file per touched data file.
6. Wire 1 through 5 into the **merge-on-read branch of `Transaction.delete()`**, gated behind `write.delete.mode=merge-on-read` on v2 tables; copy-on-write stays the default so nothing changes unless a user opts in. v1 (cannot store delete manifests) and v3 (deletion vectors, out of scope) keep the copy-on-write fallback. A size-based heuristic (rewrite tiny files instead of writing delete files) can come later.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Beginne beim merge-on-read branch von Transaction.delete() und untersuche POSITIONAL_DELETE_SCHEMA, den vorhandenen delete-file index und den v2 manifest writer. Verfolge, wie manifests und snapshots erzeugt werden, und überprüfe anschließend, dass position resolution passende data-file rows in physical order liest. Erledigt ist die Aufgabe, wenn v2 opt-in deletes exact-bound position-delete files in DELETES manifests erzeugen, während v1, v3 und das standardmäßige copy-on-write behavior unverändert bleiben.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python
Bereich
databases
Issue-Typ
Feature
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Ruhig
Klarheit
Klar beschrieben
Anfängerfreundlichkeit
38/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.