Complete deletion vector feature support
- Vorherrschende Sprache
- C++
- Sterne
- 221
- Forks
- 124
- Ø Merge
- 1 T. 16 Std.
- Gemergte PRs (30 T.)
- 21
Beschreibung
## Goal
Track the remaining work required for complete Iceberg v3 deletion-vector support in iceberg-cpp.
This is a dedicated follow-up to #637 and builds on the initial implementation in #777.
## Summary
| ID | Task | PR | Dependencies |
|---|---|---:|---|
| DV-PR-1 | Cross-language Puffin DV fixtures | [#913](https://github.com/apache/iceberg-cpp/pull/913) | — |
| DV-PR-2 | Puffin metadata validation | [#914](https://github.com/apache/iceberg-cpp/pull/914) | DV-PR-1 |
| DV-PR-3 | Scan applicability validation | [#915](https://github.com/apache/iceberg-cpp/pull/915) | DV-PR-1 |
| DV-PR-4 | Table-aware DV write pipeline | [#919](https://github.com/apache/iceberg-cpp/pull/919) | DV-PR-2, DV-PR-3 |
| DV-PR-5 | `position_deletes` metadata table | [#920](https://github.com/apache/iceberg-cpp/pull/920) | DV-PR-2 |
| DV-PR-6 | Full non-negative int64 positions | [#917](https://github.com/apache/iceberg-cpp/pull/917) | DV-PR-1 |
| DV-PR-7 | DV-aware compaction planner | [#918](https://github.com/apache/iceberg-cpp/pull/918) | DV-PR-3 |
| DV-PR-8 | DV-aware compaction executor | [#921](https://github.com/apache/iceberg-cpp/pull/921) | DV-PR-4, DV-PR-7 |
## Roadmap
### DV-PR-1 — Cross-language Puffin deletion-vector fixtures
PR: [#913](https://github.com/apache/iceberg-cpp/pull/913)
Add independently generated Go and Java Puffin fixtures covering single-blob and multi-blob deletion vectors. Verify Puffin metadata, manifest ranges, referenced data files, cardinalities, high-32-bit keys, Roaring container types, and decoded positions.
### DV-PR-2 — Validate deletion vectors against Puffin metadata
PR: [#914](https://github.com/apache/iceberg-cpp/pull/914)
Dependencies: DV-PR-1
Validate that manifest offsets and sizes identify the expected `deletion-vector-v1` Puffin blob. Check snapshot and sequence IDs, compression, referenced data file, cardinality, blob boundaries, and trailing payload data.
### DV-PR-3 — Enforce partition/spec-aware scan applicability
PR: [#915](https://github.com/apache/iceberg-cpp/pull/915)
Dependencies: DV-PR-1
Match deletion vectors to data files using the complete v3 applicability rules: referenced path, sequence number, partition spec ID, and partition tuple. Ensure applicable DVs supersede older file-scoped position deletes without applying mismatched vectors.
### DV-PR-4 — Add a table-aware v3 deletion-vector write pipeline
PR: [#919](https://github.com/apache/iceberg-cpp/pull/919)
Dependencies: DV-PR-2, DV-PR-3
Provide a table-aware position-delete update API that resolves the target data-file spec and partition, writes v3 deletion vectors, preserves the v2 Parquet path, merges existing file-scoped deletes, removes superseded entries through `RowDelta`, validates conflicts, and performs retry-safe output cleanup.
### DV-PR-5 — Add a `position_deletes` metadata table
PR: [#920](https://github.com/apache/iceberg-cpp/pull/920)
Dependencies: DV-PR-2
Expose logical position-delete rows through a general metadata table regardless of whether they are stored in Parquet files or Puffin deletion vectors. Cover partition evolution, optional deleted-row payloads, mixed upgraded tables, empty scans, and top-level projection.
### DV-PR-6 — Support all legal non-negative int64 positions
PR: [#917](https://github.com/apache/iceberg-cpp/pull/917)
Dependencies: DV-PR-1
Replace dense high-32-bit bitmap storage with sparse ordered buckets so every position from zero through `INT64_MAX` is representable without storage proportional to the largest key. Preserve wire compatibility and serialize only populated buckets in key order.
### DV-PR-7 — Add a DV-aware compaction planner
PR: [#918](https://github.com/apache/iceberg-cpp/pull/918)
Dependencies: DV-PR-3
Plan deterministic, snapshot-bound compaction groups within partition/spec boundaries. Select files using size and file-scoped delete pressure, account for DV cardinality and position deletes, exclude equality deletes, validate configuration, and shape groups around the target output size.
### DV-PR-8 — Add DV-aware compaction execution
PR: [#921](https://github.com/apache/iceberg-cpp/pull/921)
Dependencies: DV-PR-4, DV-PR-7
Execute planned groups by reading source files with all applicable deletes, writing compacted data, and atomically replacing source files while removing obsolete delete entries. Preserve row-lineage semantics, reject stale plans, protect shared Puffin objects, and clean up outputs safely across failures and unknown commit states.
## Delivery
Each item uses an independent worktree, branch, and pull request. Work is performed blockers-first, with focused tests in every PR.
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Rechercherichtung
Beginne mit der Roadmap und den Dependency-Links und behandle DV-PR-1 bis DV-PR-8 als separate Arbeitspakete statt als eine einzige Aufgabe. Im Issue werden keine Dateien oder Testpfade genannt, daher lies zuerst den relevanten verlinkten PR und dessen Dependencies. Als erledigt gilt, wenn das für das ausgewählte Arbeitspaket abgegrenzte Verhalten mit fokussierten Tests in dessen unabhängigem Worktree, Branch und Pull Request implementiert ist.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- cpp
- Bereich
- databases
- Issue-Typ
- Feature
- Schwierigkeit
- 5/5
- Geschätzter Aufwand
- Über eine Woche
- Aktivitätsstatus
- Veraltet
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 20/100