Complete deletion vector feature support
- Lenguaje dominante
- C++
- Estrellas
- 221
- Forks
- 124
- Merge medio
- 1 d 16 h
- PR fusionados (30 d)
- 21
Descripción
## Goal
Track the remaining work required for complete Iceberg v3 deletion-vector support in iceberg-cpp.
This is a dedicated follow-up to #637 and builds on the initial implementation in #777.
## Summary
| ID | Task | PR | Dependencies |
|---|---|---:|---|
| DV-PR-1 | Cross-language Puffin DV fixtures | [#913](https://github.com/apache/iceberg-cpp/pull/913) | — |
| DV-PR-2 | Puffin metadata validation | [#914](https://github.com/apache/iceberg-cpp/pull/914) | DV-PR-1 |
| DV-PR-3 | Scan applicability validation | [#915](https://github.com/apache/iceberg-cpp/pull/915) | DV-PR-1 |
| DV-PR-4 | Table-aware DV write pipeline | [#919](https://github.com/apache/iceberg-cpp/pull/919) | DV-PR-2, DV-PR-3 |
| DV-PR-5 | `position_deletes` metadata table | [#920](https://github.com/apache/iceberg-cpp/pull/920) | DV-PR-2 |
| DV-PR-6 | Full non-negative int64 positions | [#917](https://github.com/apache/iceberg-cpp/pull/917) | DV-PR-1 |
| DV-PR-7 | DV-aware compaction planner | [#918](https://github.com/apache/iceberg-cpp/pull/918) | DV-PR-3 |
| DV-PR-8 | DV-aware compaction executor | [#921](https://github.com/apache/iceberg-cpp/pull/921) | DV-PR-4, DV-PR-7 |
## Roadmap
### DV-PR-1 — Cross-language Puffin deletion-vector fixtures
PR: [#913](https://github.com/apache/iceberg-cpp/pull/913)
Add independently generated Go and Java Puffin fixtures covering single-blob and multi-blob deletion vectors. Verify Puffin metadata, manifest ranges, referenced data files, cardinalities, high-32-bit keys, Roaring container types, and decoded positions.
### DV-PR-2 — Validate deletion vectors against Puffin metadata
PR: [#914](https://github.com/apache/iceberg-cpp/pull/914)
Dependencies: DV-PR-1
Validate that manifest offsets and sizes identify the expected `deletion-vector-v1` Puffin blob. Check snapshot and sequence IDs, compression, referenced data file, cardinality, blob boundaries, and trailing payload data.
### DV-PR-3 — Enforce partition/spec-aware scan applicability
PR: [#915](https://github.com/apache/iceberg-cpp/pull/915)
Dependencies: DV-PR-1
Match deletion vectors to data files using the complete v3 applicability rules: referenced path, sequence number, partition spec ID, and partition tuple. Ensure applicable DVs supersede older file-scoped position deletes without applying mismatched vectors.
### DV-PR-4 — Add a table-aware v3 deletion-vector write pipeline
PR: [#919](https://github.com/apache/iceberg-cpp/pull/919)
Dependencies: DV-PR-2, DV-PR-3
Provide a table-aware position-delete update API that resolves the target data-file spec and partition, writes v3 deletion vectors, preserves the v2 Parquet path, merges existing file-scoped deletes, removes superseded entries through `RowDelta`, validates conflicts, and performs retry-safe output cleanup.
### DV-PR-5 — Add a `position_deletes` metadata table
PR: [#920](https://github.com/apache/iceberg-cpp/pull/920)
Dependencies: DV-PR-2
Expose logical position-delete rows through a general metadata table regardless of whether they are stored in Parquet files or Puffin deletion vectors. Cover partition evolution, optional deleted-row payloads, mixed upgraded tables, empty scans, and top-level projection.
### DV-PR-6 — Support all legal non-negative int64 positions
PR: [#917](https://github.com/apache/iceberg-cpp/pull/917)
Dependencies: DV-PR-1
Replace dense high-32-bit bitmap storage with sparse ordered buckets so every position from zero through `INT64_MAX` is representable without storage proportional to the largest key. Preserve wire compatibility and serialize only populated buckets in key order.
### DV-PR-7 — Add a DV-aware compaction planner
PR: [#918](https://github.com/apache/iceberg-cpp/pull/918)
Dependencies: DV-PR-3
Plan deterministic, snapshot-bound compaction groups within partition/spec boundaries. Select files using size and file-scoped delete pressure, account for DV cardinality and position deletes, exclude equality deletes, validate configuration, and shape groups around the target output size.
### DV-PR-8 — Add DV-aware compaction execution
PR: [#921](https://github.com/apache/iceberg-cpp/pull/921)
Dependencies: DV-PR-4, DV-PR-7
Execute planned groups by reading source files with all applicable deletes, writing compacted data, and atomically replacing source files while removing obsolete delete entries. Preserve row-lineage semantics, reject stale plans, protect shared Puffin objects, and clean up outputs safely across failures and unknown commit states.
## Delivery
Each item uses an independent worktree, branch, and pull request. Work is performed blockers-first, with focused tests in every PR.
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Línea de trabajo
Comienza con la hoja de ruta y los enlaces de dependencias, tratando DV-PR-1 a DV-PR-8 como elementos de trabajo separados en lugar de una sola tarea. El issue no indica archivos ni rutas de tests, así que lee primero el PR enlazado relevante y sus dependencias. Se considera terminado cuando el comportamiento delimitado del elemento seleccionado está implementado con tests específicos en su worktree, branch y pull request independientes.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- cpp
- Área
- databases
- Tipo de issue
- Nueva funcionalidad
- Dificultad
- 5/5
- Tiempo estimado
- Más de una semana
- Estado de actividad
- Estancado
- Claridad
- Bastante claro
- Aptitud para principiantes
- 20/100