Complete deletion vector feature support
- 主要语言
- C++
- 星标
- 221
- 派生
- 124
- 平均合并
- 1 天 16 小时
- 30 天内合并 PR
- 21
描述
## Goal
Track the remaining work required for complete Iceberg v3 deletion-vector support in iceberg-cpp.
This is a dedicated follow-up to #637 and builds on the initial implementation in #777.
## Summary
| ID | Task | PR | Dependencies |
|---|---|---:|---|
| DV-PR-1 | Cross-language Puffin DV fixtures | [#913](https://github.com/apache/iceberg-cpp/pull/913) | — |
| DV-PR-2 | Puffin metadata validation | [#914](https://github.com/apache/iceberg-cpp/pull/914) | DV-PR-1 |
| DV-PR-3 | Scan applicability validation | [#915](https://github.com/apache/iceberg-cpp/pull/915) | DV-PR-1 |
| DV-PR-4 | Table-aware DV write pipeline | [#919](https://github.com/apache/iceberg-cpp/pull/919) | DV-PR-2, DV-PR-3 |
| DV-PR-5 | `position_deletes` metadata table | [#920](https://github.com/apache/iceberg-cpp/pull/920) | DV-PR-2 |
| DV-PR-6 | Full non-negative int64 positions | [#917](https://github.com/apache/iceberg-cpp/pull/917) | DV-PR-1 |
| DV-PR-7 | DV-aware compaction planner | [#918](https://github.com/apache/iceberg-cpp/pull/918) | DV-PR-3 |
| DV-PR-8 | DV-aware compaction executor | [#921](https://github.com/apache/iceberg-cpp/pull/921) | DV-PR-4, DV-PR-7 |
## Roadmap
### DV-PR-1 — Cross-language Puffin deletion-vector fixtures
PR: [#913](https://github.com/apache/iceberg-cpp/pull/913)
Add independently generated Go and Java Puffin fixtures covering single-blob and multi-blob deletion vectors. Verify Puffin metadata, manifest ranges, referenced data files, cardinalities, high-32-bit keys, Roaring container types, and decoded positions.
### DV-PR-2 — Validate deletion vectors against Puffin metadata
PR: [#914](https://github.com/apache/iceberg-cpp/pull/914)
Dependencies: DV-PR-1
Validate that manifest offsets and sizes identify the expected `deletion-vector-v1` Puffin blob. Check snapshot and sequence IDs, compression, referenced data file, cardinality, blob boundaries, and trailing payload data.
### DV-PR-3 — Enforce partition/spec-aware scan applicability
PR: [#915](https://github.com/apache/iceberg-cpp/pull/915)
Dependencies: DV-PR-1
Match deletion vectors to data files using the complete v3 applicability rules: referenced path, sequence number, partition spec ID, and partition tuple. Ensure applicable DVs supersede older file-scoped position deletes without applying mismatched vectors.
### DV-PR-4 — Add a table-aware v3 deletion-vector write pipeline
PR: [#919](https://github.com/apache/iceberg-cpp/pull/919)
Dependencies: DV-PR-2, DV-PR-3
Provide a table-aware position-delete update API that resolves the target data-file spec and partition, writes v3 deletion vectors, preserves the v2 Parquet path, merges existing file-scoped deletes, removes superseded entries through `RowDelta`, validates conflicts, and performs retry-safe output cleanup.
### DV-PR-5 — Add a `position_deletes` metadata table
PR: [#920](https://github.com/apache/iceberg-cpp/pull/920)
Dependencies: DV-PR-2
Expose logical position-delete rows through a general metadata table regardless of whether they are stored in Parquet files or Puffin deletion vectors. Cover partition evolution, optional deleted-row payloads, mixed upgraded tables, empty scans, and top-level projection.
### DV-PR-6 — Support all legal non-negative int64 positions
PR: [#917](https://github.com/apache/iceberg-cpp/pull/917)
Dependencies: DV-PR-1
Replace dense high-32-bit bitmap storage with sparse ordered buckets so every position from zero through `INT64_MAX` is representable without storage proportional to the largest key. Preserve wire compatibility and serialize only populated buckets in key order.
### DV-PR-7 — Add a DV-aware compaction planner
PR: [#918](https://github.com/apache/iceberg-cpp/pull/918)
Dependencies: DV-PR-3
Plan deterministic, snapshot-bound compaction groups within partition/spec boundaries. Select files using size and file-scoped delete pressure, account for DV cardinality and position deletes, exclude equality deletes, validate configuration, and shape groups around the target output size.
### DV-PR-8 — Add DV-aware compaction execution
PR: [#921](https://github.com/apache/iceberg-cpp/pull/921)
Dependencies: DV-PR-4, DV-PR-7
Execute planned groups by reading source files with all applicable deletes, writing compacted data, and atomically replacing source files while removing obsolete delete entries. Preserve row-lineage semantics, reject stale plans, protect shared Puffin objects, and clean up outputs safely across failures and unknown commit states.
## Delivery
Each item uses an independent worktree, branch, and pull request. Work is performed blockers-first, with focused tests in every PR.
贡献指南
这个仓库没有索引到贡献指南
调研方向
从 roadmap 和依赖链接开始,将 DV-PR-1 到 DV-PR-8 视为彼此独立的工作项,而不是一个任务。issue 未列出文件或测试路径,因此请先阅读相关的已链接 PR 及其依赖项。完成的标准是:在所选工作项独立的 worktree、branch 和 pull request 中,通过有针对性的测试实现该工作项范围内的行为。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- cpp
- 领域
- databases
- Issue 类型
- 功能
- 难度
- 5/5
- 预计耗时
- 一周以上
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 20/100