apache / apache/iceberg-cpp

Complete deletion vector feature support

オープン
#916 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
C++
スター
221
フォーク
124
平均マージ
1日 16時間
マージ済み PR(30日)
21

説明

## Goal

Track the remaining work required for complete Iceberg v3 deletion-vector support in iceberg-cpp.

This is a dedicated follow-up to #637 and builds on the initial implementation in #777.

## Summary

| ID | Task | PR | Dependencies |
|---|---|---:|---|
| DV-PR-1 | Cross-language Puffin DV fixtures | [#913](https://github.com/apache/iceberg-cpp/pull/913) | — |
| DV-PR-2 | Puffin metadata validation | [#914](https://github.com/apache/iceberg-cpp/pull/914) | DV-PR-1 |
| DV-PR-3 | Scan applicability validation | [#915](https://github.com/apache/iceberg-cpp/pull/915) | DV-PR-1 |
| DV-PR-4 | Table-aware DV write pipeline | [#919](https://github.com/apache/iceberg-cpp/pull/919) | DV-PR-2, DV-PR-3 |
| DV-PR-5 | `position_deletes` metadata table | [#920](https://github.com/apache/iceberg-cpp/pull/920) | DV-PR-2 |
| DV-PR-6 | Full non-negative int64 positions | [#917](https://github.com/apache/iceberg-cpp/pull/917) | DV-PR-1 |
| DV-PR-7 | DV-aware compaction planner | [#918](https://github.com/apache/iceberg-cpp/pull/918) | DV-PR-3 |
| DV-PR-8 | DV-aware compaction executor | [#921](https://github.com/apache/iceberg-cpp/pull/921) | DV-PR-4, DV-PR-7 |

## Roadmap

### DV-PR-1 — Cross-language Puffin deletion-vector fixtures

PR: [#913](https://github.com/apache/iceberg-cpp/pull/913)

Add independently generated Go and Java Puffin fixtures covering single-blob and multi-blob deletion vectors. Verify Puffin metadata, manifest ranges, referenced data files, cardinalities, high-32-bit keys, Roaring container types, and decoded positions.

### DV-PR-2 — Validate deletion vectors against Puffin metadata

PR: [#914](https://github.com/apache/iceberg-cpp/pull/914)
Dependencies: DV-PR-1

Validate that manifest offsets and sizes identify the expected `deletion-vector-v1` Puffin blob. Check snapshot and sequence IDs, compression, referenced data file, cardinality, blob boundaries, and trailing payload data.

### DV-PR-3 — Enforce partition/spec-aware scan applicability

PR: [#915](https://github.com/apache/iceberg-cpp/pull/915)
Dependencies: DV-PR-1

Match deletion vectors to data files using the complete v3 applicability rules: referenced path, sequence number, partition spec ID, and partition tuple. Ensure applicable DVs supersede older file-scoped position deletes without applying mismatched vectors.

### DV-PR-4 — Add a table-aware v3 deletion-vector write pipeline

PR: [#919](https://github.com/apache/iceberg-cpp/pull/919)
Dependencies: DV-PR-2, DV-PR-3

Provide a table-aware position-delete update API that resolves the target data-file spec and partition, writes v3 deletion vectors, preserves the v2 Parquet path, merges existing file-scoped deletes, removes superseded entries through `RowDelta`, validates conflicts, and performs retry-safe output cleanup.

### DV-PR-5 — Add a `position_deletes` metadata table

PR: [#920](https://github.com/apache/iceberg-cpp/pull/920)
Dependencies: DV-PR-2

Expose logical position-delete rows through a general metadata table regardless of whether they are stored in Parquet files or Puffin deletion vectors. Cover partition evolution, optional deleted-row payloads, mixed upgraded tables, empty scans, and top-level projection.

### DV-PR-6 — Support all legal non-negative int64 positions

PR: [#917](https://github.com/apache/iceberg-cpp/pull/917)
Dependencies: DV-PR-1

Replace dense high-32-bit bitmap storage with sparse ordered buckets so every position from zero through `INT64_MAX` is representable without storage proportional to the largest key. Preserve wire compatibility and serialize only populated buckets in key order.

### DV-PR-7 — Add a DV-aware compaction planner

PR: [#918](https://github.com/apache/iceberg-cpp/pull/918)
Dependencies: DV-PR-3

Plan deterministic, snapshot-bound compaction groups within partition/spec boundaries. Select files using size and file-scoped delete pressure, account for DV cardinality and position deletes, exclude equality deletes, validate configuration, and shape groups around the target output size.

### DV-PR-8 — Add DV-aware compaction execution

PR: [#921](https://github.com/apache/iceberg-cpp/pull/921)
Dependencies: DV-PR-4, DV-PR-7

Execute planned groups by reading source files with all applicable deletes, writing compacted data, and atomically replacing source files while removing obsolete delete entries. Preserve row-lineage semantics, reject stale plans, protect shared Puffin objects, and clean up outputs safely across failures and unknown commit states.

## Delivery

Each item uses an independent worktree, branch, and pull request. Work is performed blockers-first, with focused tests in every PR.

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

調査の方向性

ロードマップと依存関係のリンクから始め、DV-PR-1 から DV-PR-8 を 1 つのタスクではなく個別の作業項目として扱います。issue にはファイルやテストパスが記載されていないため、まず関連するリンク先の PR とその依存関係を確認してください。完了とは、選択した作業項目のスコープに含まれる動作が、独立した worktree、branch、pull request で、焦点を絞ったテストとともに実装されていることを意味します。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
cpp
領域
databases
issue の種類
機能追加
難易度
5/5
見積もり時間
1週間以上
活発さ
停滞
明瞭さ
おおむね明確
初心者へのやさしさ
20/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。