Complete deletion vector feature support
- Dominant language
- C++
- Stars
- 221
- Forks
- 124
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 21
Description
## Goal
Track the remaining work required for complete Iceberg v3 deletion-vector support in iceberg-cpp.
This is a dedicated follow-up to #637 and builds on the initial implementation in #777.
## Summary
| ID | Task | PR | Dependencies |
|---|---|---:|---|
| DV-PR-1 | Cross-language Puffin DV fixtures | [#913](https://github.com/apache/iceberg-cpp/pull/913) | — |
| DV-PR-2 | Puffin metadata validation | [#914](https://github.com/apache/iceberg-cpp/pull/914) | DV-PR-1 |
| DV-PR-3 | Scan applicability validation | [#915](https://github.com/apache/iceberg-cpp/pull/915) | DV-PR-1 |
| DV-PR-4 | Table-aware DV write pipeline | [#919](https://github.com/apache/iceberg-cpp/pull/919) | DV-PR-2, DV-PR-3 |
| DV-PR-5 | `position_deletes` metadata table | [#920](https://github.com/apache/iceberg-cpp/pull/920) | DV-PR-2 |
| DV-PR-6 | Full non-negative int64 positions | [#917](https://github.com/apache/iceberg-cpp/pull/917) | DV-PR-1 |
| DV-PR-7 | DV-aware compaction planner | [#918](https://github.com/apache/iceberg-cpp/pull/918) | DV-PR-3 |
| DV-PR-8 | DV-aware compaction executor | [#921](https://github.com/apache/iceberg-cpp/pull/921) | DV-PR-4, DV-PR-7 |
## Roadmap
### DV-PR-1 — Cross-language Puffin deletion-vector fixtures
PR: [#913](https://github.com/apache/iceberg-cpp/pull/913)
Add independently generated Go and Java Puffin fixtures covering single-blob and multi-blob deletion vectors. Verify Puffin metadata, manifest ranges, referenced data files, cardinalities, high-32-bit keys, Roaring container types, and decoded positions.
### DV-PR-2 — Validate deletion vectors against Puffin metadata
PR: [#914](https://github.com/apache/iceberg-cpp/pull/914)
Dependencies: DV-PR-1
Validate that manifest offsets and sizes identify the expected `deletion-vector-v1` Puffin blob. Check snapshot and sequence IDs, compression, referenced data file, cardinality, blob boundaries, and trailing payload data.
### DV-PR-3 — Enforce partition/spec-aware scan applicability
PR: [#915](https://github.com/apache/iceberg-cpp/pull/915)
Dependencies: DV-PR-1
Match deletion vectors to data files using the complete v3 applicability rules: referenced path, sequence number, partition spec ID, and partition tuple. Ensure applicable DVs supersede older file-scoped position deletes without applying mismatched vectors.
### DV-PR-4 — Add a table-aware v3 deletion-vector write pipeline
PR: [#919](https://github.com/apache/iceberg-cpp/pull/919)
Dependencies: DV-PR-2, DV-PR-3
Provide a table-aware position-delete update API that resolves the target data-file spec and partition, writes v3 deletion vectors, preserves the v2 Parquet path, merges existing file-scoped deletes, removes superseded entries through `RowDelta`, validates conflicts, and performs retry-safe output cleanup.
### DV-PR-5 — Add a `position_deletes` metadata table
PR: [#920](https://github.com/apache/iceberg-cpp/pull/920)
Dependencies: DV-PR-2
Expose logical position-delete rows through a general metadata table regardless of whether they are stored in Parquet files or Puffin deletion vectors. Cover partition evolution, optional deleted-row payloads, mixed upgraded tables, empty scans, and top-level projection.
### DV-PR-6 — Support all legal non-negative int64 positions
PR: [#917](https://github.com/apache/iceberg-cpp/pull/917)
Dependencies: DV-PR-1
Replace dense high-32-bit bitmap storage with sparse ordered buckets so every position from zero through `INT64_MAX` is representable without storage proportional to the largest key. Preserve wire compatibility and serialize only populated buckets in key order.
### DV-PR-7 — Add a DV-aware compaction planner
PR: [#918](https://github.com/apache/iceberg-cpp/pull/918)
Dependencies: DV-PR-3
Plan deterministic, snapshot-bound compaction groups within partition/spec boundaries. Select files using size and file-scoped delete pressure, account for DV cardinality and position deletes, exclude equality deletes, validate configuration, and shape groups around the target output size.
### DV-PR-8 — Add DV-aware compaction execution
PR: [#921](https://github.com/apache/iceberg-cpp/pull/921)
Dependencies: DV-PR-4, DV-PR-7
Execute planned groups by reading source files with all applicable deletes, writing compacted data, and atomically replacing source files while removing obsolete delete entries. Preserve row-lineage semantics, reject stale plans, protect shared Puffin objects, and clean up outputs safely across failures and unknown commit states.
## Delivery
Each item uses an independent worktree, branch, and pull request. Work is performed blockers-first, with focused tests in every PR.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the roadmap and dependency links, treating DV-PR-1 through DV-PR-8 as separate work items rather than one task. The issue names no files or test paths, so read the relevant linked PR and its dependencies first. Done means the selected item's scoped behavior is implemented with focused tests in its independent worktree, branch, and pull request.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100