apache / apache/iceberg-cpp

Complete deletion vector feature support

Ouverte
#916 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
C++
Étoiles
221
Forks
124
Merge moyen
1 j 16 h
PR mergées (30 j)
21

Description

## Goal

Track the remaining work required for complete Iceberg v3 deletion-vector support in iceberg-cpp.

This is a dedicated follow-up to #637 and builds on the initial implementation in #777.

## Summary

| ID | Task | PR | Dependencies |
|---|---|---:|---|
| DV-PR-1 | Cross-language Puffin DV fixtures | [#913](https://github.com/apache/iceberg-cpp/pull/913) | — |
| DV-PR-2 | Puffin metadata validation | [#914](https://github.com/apache/iceberg-cpp/pull/914) | DV-PR-1 |
| DV-PR-3 | Scan applicability validation | [#915](https://github.com/apache/iceberg-cpp/pull/915) | DV-PR-1 |
| DV-PR-4 | Table-aware DV write pipeline | [#919](https://github.com/apache/iceberg-cpp/pull/919) | DV-PR-2, DV-PR-3 |
| DV-PR-5 | `position_deletes` metadata table | [#920](https://github.com/apache/iceberg-cpp/pull/920) | DV-PR-2 |
| DV-PR-6 | Full non-negative int64 positions | [#917](https://github.com/apache/iceberg-cpp/pull/917) | DV-PR-1 |
| DV-PR-7 | DV-aware compaction planner | [#918](https://github.com/apache/iceberg-cpp/pull/918) | DV-PR-3 |
| DV-PR-8 | DV-aware compaction executor | [#921](https://github.com/apache/iceberg-cpp/pull/921) | DV-PR-4, DV-PR-7 |

## Roadmap

### DV-PR-1 — Cross-language Puffin deletion-vector fixtures

PR: [#913](https://github.com/apache/iceberg-cpp/pull/913)

Add independently generated Go and Java Puffin fixtures covering single-blob and multi-blob deletion vectors. Verify Puffin metadata, manifest ranges, referenced data files, cardinalities, high-32-bit keys, Roaring container types, and decoded positions.

### DV-PR-2 — Validate deletion vectors against Puffin metadata

PR: [#914](https://github.com/apache/iceberg-cpp/pull/914)
Dependencies: DV-PR-1

Validate that manifest offsets and sizes identify the expected `deletion-vector-v1` Puffin blob. Check snapshot and sequence IDs, compression, referenced data file, cardinality, blob boundaries, and trailing payload data.

### DV-PR-3 — Enforce partition/spec-aware scan applicability

PR: [#915](https://github.com/apache/iceberg-cpp/pull/915)
Dependencies: DV-PR-1

Match deletion vectors to data files using the complete v3 applicability rules: referenced path, sequence number, partition spec ID, and partition tuple. Ensure applicable DVs supersede older file-scoped position deletes without applying mismatched vectors.

### DV-PR-4 — Add a table-aware v3 deletion-vector write pipeline

PR: [#919](https://github.com/apache/iceberg-cpp/pull/919)
Dependencies: DV-PR-2, DV-PR-3

Provide a table-aware position-delete update API that resolves the target data-file spec and partition, writes v3 deletion vectors, preserves the v2 Parquet path, merges existing file-scoped deletes, removes superseded entries through `RowDelta`, validates conflicts, and performs retry-safe output cleanup.

### DV-PR-5 — Add a `position_deletes` metadata table

PR: [#920](https://github.com/apache/iceberg-cpp/pull/920)
Dependencies: DV-PR-2

Expose logical position-delete rows through a general metadata table regardless of whether they are stored in Parquet files or Puffin deletion vectors. Cover partition evolution, optional deleted-row payloads, mixed upgraded tables, empty scans, and top-level projection.

### DV-PR-6 — Support all legal non-negative int64 positions

PR: [#917](https://github.com/apache/iceberg-cpp/pull/917)
Dependencies: DV-PR-1

Replace dense high-32-bit bitmap storage with sparse ordered buckets so every position from zero through `INT64_MAX` is representable without storage proportional to the largest key. Preserve wire compatibility and serialize only populated buckets in key order.

### DV-PR-7 — Add a DV-aware compaction planner

PR: [#918](https://github.com/apache/iceberg-cpp/pull/918)
Dependencies: DV-PR-3

Plan deterministic, snapshot-bound compaction groups within partition/spec boundaries. Select files using size and file-scoped delete pressure, account for DV cardinality and position deletes, exclude equality deletes, validate configuration, and shape groups around the target output size.

### DV-PR-8 — Add DV-aware compaction execution

PR: [#921](https://github.com/apache/iceberg-cpp/pull/921)
Dependencies: DV-PR-4, DV-PR-7

Execute planned groups by reading source files with all applicable deletes, writing compacted data, and atomically replacing source files while removing obsolete delete entries. Preserve row-lineage semantics, reject stale plans, protect shared Puffin objects, and clean up outputs safely across failures and unknown commit states.

## Delivery

Each item uses an independent worktree, branch, and pull request. Work is performed blockers-first, with focused tests in every PR.

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Piste de recherche

Commencez par la roadmap et les liens de dépendances, en traitant DV-PR-1 à DV-PR-8 comme des éléments de travail distincts plutôt que comme une seule tâche. L’issue ne mentionne aucun fichier ni chemin de test ; lisez donc d’abord le PR pertinent lié et ses dépendances. Le travail est considéré comme terminé lorsque le comportement défini pour l’élément sélectionné est implémenté avec des tests ciblés dans son worktree, sa branche et sa pull request indépendants.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
cpp
Domaine
databases
Type d'issue
Fonctionnalité
Difficulté
5/5
Temps estimé
Plus d'une semaine
Activité
À l'abandon
Clarté
Plutôt claire
Accessibilité débutants
20/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.