apache / apache/iceberg-cpp

Complete deletion vector feature support

Open
#916 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
221
Forks
124
Avg merge
1d 16h
Merged PRs (30d)
21

Description

## Goal

Track the remaining work required for complete Iceberg v3 deletion-vector support in iceberg-cpp.

This is a dedicated follow-up to #637 and builds on the initial implementation in #777.

## Summary

| ID | Task | PR | Dependencies |
|---|---|---:|---|
| DV-PR-1 | Cross-language Puffin DV fixtures | [#913](https://github.com/apache/iceberg-cpp/pull/913) | — |
| DV-PR-2 | Puffin metadata validation | [#914](https://github.com/apache/iceberg-cpp/pull/914) | DV-PR-1 |
| DV-PR-3 | Scan applicability validation | [#915](https://github.com/apache/iceberg-cpp/pull/915) | DV-PR-1 |
| DV-PR-4 | Table-aware DV write pipeline | [#919](https://github.com/apache/iceberg-cpp/pull/919) | DV-PR-2, DV-PR-3 |
| DV-PR-5 | `position_deletes` metadata table | [#920](https://github.com/apache/iceberg-cpp/pull/920) | DV-PR-2 |
| DV-PR-6 | Full non-negative int64 positions | [#917](https://github.com/apache/iceberg-cpp/pull/917) | DV-PR-1 |
| DV-PR-7 | DV-aware compaction planner | [#918](https://github.com/apache/iceberg-cpp/pull/918) | DV-PR-3 |
| DV-PR-8 | DV-aware compaction executor | [#921](https://github.com/apache/iceberg-cpp/pull/921) | DV-PR-4, DV-PR-7 |

## Roadmap

### DV-PR-1 — Cross-language Puffin deletion-vector fixtures

PR: [#913](https://github.com/apache/iceberg-cpp/pull/913)

Add independently generated Go and Java Puffin fixtures covering single-blob and multi-blob deletion vectors. Verify Puffin metadata, manifest ranges, referenced data files, cardinalities, high-32-bit keys, Roaring container types, and decoded positions.

### DV-PR-2 — Validate deletion vectors against Puffin metadata

PR: [#914](https://github.com/apache/iceberg-cpp/pull/914)
Dependencies: DV-PR-1

Validate that manifest offsets and sizes identify the expected `deletion-vector-v1` Puffin blob. Check snapshot and sequence IDs, compression, referenced data file, cardinality, blob boundaries, and trailing payload data.

### DV-PR-3 — Enforce partition/spec-aware scan applicability

PR: [#915](https://github.com/apache/iceberg-cpp/pull/915)
Dependencies: DV-PR-1

Match deletion vectors to data files using the complete v3 applicability rules: referenced path, sequence number, partition spec ID, and partition tuple. Ensure applicable DVs supersede older file-scoped position deletes without applying mismatched vectors.

### DV-PR-4 — Add a table-aware v3 deletion-vector write pipeline

PR: [#919](https://github.com/apache/iceberg-cpp/pull/919)
Dependencies: DV-PR-2, DV-PR-3

Provide a table-aware position-delete update API that resolves the target data-file spec and partition, writes v3 deletion vectors, preserves the v2 Parquet path, merges existing file-scoped deletes, removes superseded entries through `RowDelta`, validates conflicts, and performs retry-safe output cleanup.

### DV-PR-5 — Add a `position_deletes` metadata table

PR: [#920](https://github.com/apache/iceberg-cpp/pull/920)
Dependencies: DV-PR-2

Expose logical position-delete rows through a general metadata table regardless of whether they are stored in Parquet files or Puffin deletion vectors. Cover partition evolution, optional deleted-row payloads, mixed upgraded tables, empty scans, and top-level projection.

### DV-PR-6 — Support all legal non-negative int64 positions

PR: [#917](https://github.com/apache/iceberg-cpp/pull/917)
Dependencies: DV-PR-1

Replace dense high-32-bit bitmap storage with sparse ordered buckets so every position from zero through `INT64_MAX` is representable without storage proportional to the largest key. Preserve wire compatibility and serialize only populated buckets in key order.

### DV-PR-7 — Add a DV-aware compaction planner

PR: [#918](https://github.com/apache/iceberg-cpp/pull/918)
Dependencies: DV-PR-3

Plan deterministic, snapshot-bound compaction groups within partition/spec boundaries. Select files using size and file-scoped delete pressure, account for DV cardinality and position deletes, exclude equality deletes, validate configuration, and shape groups around the target output size.

### DV-PR-8 — Add DV-aware compaction execution

PR: [#921](https://github.com/apache/iceberg-cpp/pull/921)
Dependencies: DV-PR-4, DV-PR-7

Execute planned groups by reading source files with all applicable deletes, writing compacted data, and atomically replacing source files while removing obsolete delete entries. Preserve row-lineage semantics, reject stale plans, protect shared Puffin objects, and clean up outputs safely across failures and unknown commit states.

## Delivery

Each item uses an independent worktree, branch, and pull request. Work is performed blockers-first, with focused tests in every PR.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the roadmap and dependency links, treating DV-PR-1 through DV-PR-8 as separate work items rather than one task. The issue names no files or test paths, so read the relevant linked PR and its dependencies first. Done means the selected item's scoped behavior is implemented with focused tests in its independent worktree, branch, and pull request.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.