apache / apache/iceberg-cpp

Complete deletion vector feature support

未关闭
#916 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
C++
星标
221
派生
124
平均合并
1 天 16 小时
30 天内合并 PR
21

描述

## Goal

Track the remaining work required for complete Iceberg v3 deletion-vector support in iceberg-cpp.

This is a dedicated follow-up to #637 and builds on the initial implementation in #777.

## Summary

| ID | Task | PR | Dependencies |
|---|---|---:|---|
| DV-PR-1 | Cross-language Puffin DV fixtures | [#913](https://github.com/apache/iceberg-cpp/pull/913) | — |
| DV-PR-2 | Puffin metadata validation | [#914](https://github.com/apache/iceberg-cpp/pull/914) | DV-PR-1 |
| DV-PR-3 | Scan applicability validation | [#915](https://github.com/apache/iceberg-cpp/pull/915) | DV-PR-1 |
| DV-PR-4 | Table-aware DV write pipeline | [#919](https://github.com/apache/iceberg-cpp/pull/919) | DV-PR-2, DV-PR-3 |
| DV-PR-5 | `position_deletes` metadata table | [#920](https://github.com/apache/iceberg-cpp/pull/920) | DV-PR-2 |
| DV-PR-6 | Full non-negative int64 positions | [#917](https://github.com/apache/iceberg-cpp/pull/917) | DV-PR-1 |
| DV-PR-7 | DV-aware compaction planner | [#918](https://github.com/apache/iceberg-cpp/pull/918) | DV-PR-3 |
| DV-PR-8 | DV-aware compaction executor | [#921](https://github.com/apache/iceberg-cpp/pull/921) | DV-PR-4, DV-PR-7 |

## Roadmap

### DV-PR-1 — Cross-language Puffin deletion-vector fixtures

PR: [#913](https://github.com/apache/iceberg-cpp/pull/913)

Add independently generated Go and Java Puffin fixtures covering single-blob and multi-blob deletion vectors. Verify Puffin metadata, manifest ranges, referenced data files, cardinalities, high-32-bit keys, Roaring container types, and decoded positions.

### DV-PR-2 — Validate deletion vectors against Puffin metadata

PR: [#914](https://github.com/apache/iceberg-cpp/pull/914)
Dependencies: DV-PR-1

Validate that manifest offsets and sizes identify the expected `deletion-vector-v1` Puffin blob. Check snapshot and sequence IDs, compression, referenced data file, cardinality, blob boundaries, and trailing payload data.

### DV-PR-3 — Enforce partition/spec-aware scan applicability

PR: [#915](https://github.com/apache/iceberg-cpp/pull/915)
Dependencies: DV-PR-1

Match deletion vectors to data files using the complete v3 applicability rules: referenced path, sequence number, partition spec ID, and partition tuple. Ensure applicable DVs supersede older file-scoped position deletes without applying mismatched vectors.

### DV-PR-4 — Add a table-aware v3 deletion-vector write pipeline

PR: [#919](https://github.com/apache/iceberg-cpp/pull/919)
Dependencies: DV-PR-2, DV-PR-3

Provide a table-aware position-delete update API that resolves the target data-file spec and partition, writes v3 deletion vectors, preserves the v2 Parquet path, merges existing file-scoped deletes, removes superseded entries through `RowDelta`, validates conflicts, and performs retry-safe output cleanup.

### DV-PR-5 — Add a `position_deletes` metadata table

PR: [#920](https://github.com/apache/iceberg-cpp/pull/920)
Dependencies: DV-PR-2

Expose logical position-delete rows through a general metadata table regardless of whether they are stored in Parquet files or Puffin deletion vectors. Cover partition evolution, optional deleted-row payloads, mixed upgraded tables, empty scans, and top-level projection.

### DV-PR-6 — Support all legal non-negative int64 positions

PR: [#917](https://github.com/apache/iceberg-cpp/pull/917)
Dependencies: DV-PR-1

Replace dense high-32-bit bitmap storage with sparse ordered buckets so every position from zero through `INT64_MAX` is representable without storage proportional to the largest key. Preserve wire compatibility and serialize only populated buckets in key order.

### DV-PR-7 — Add a DV-aware compaction planner

PR: [#918](https://github.com/apache/iceberg-cpp/pull/918)
Dependencies: DV-PR-3

Plan deterministic, snapshot-bound compaction groups within partition/spec boundaries. Select files using size and file-scoped delete pressure, account for DV cardinality and position deletes, exclude equality deletes, validate configuration, and shape groups around the target output size.

### DV-PR-8 — Add DV-aware compaction execution

PR: [#921](https://github.com/apache/iceberg-cpp/pull/921)
Dependencies: DV-PR-4, DV-PR-7

Execute planned groups by reading source files with all applicable deletes, writing compacted data, and atomically replacing source files while removing obsolete delete entries. Preserve row-lineage semantics, reject stale plans, protect shared Puffin objects, and clean up outputs safely across failures and unknown commit states.

## Delivery

Each item uses an independent worktree, branch, and pull request. Work is performed blockers-first, with focused tests in every PR.

贡献指南

这个仓库没有索引到贡献指南

调研方向

从 roadmap 和依赖链接开始,将 DV-PR-1 到 DV-PR-8 视为彼此独立的工作项,而不是一个任务。issue 未列出文件或测试路径,因此请先阅读相关的已链接 PR 及其依赖项。完成的标准是:在所选工作项独立的 worktree、branch 和 pull request 中,通过有针对性的测试实现该工作项范围内的行为。

由索引模型根据 Issue 内容生成。

评估

技术栈
cpp
领域
databases
Issue 类型
功能
难度
5/5
预计耗时
一周以上
活跃度
停滞
描述清晰度
基本清楚
新手友好度
20/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。