apache / apache/gluten

[VL] Support Iceberg V3 deletion vectors in Velox

Open
#12,628 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Description

Add native Iceberg format-version 3 deletion-vector support to the Velox backend.

Today Gluten falls back when reading Puffin deletion vectors and row-level DELETE, UPDATE, and MERGE operations do not use the enhanced native Iceberg writer. Velox already contains the required deletion-vector writer support.

Proposed scope:

- Pass Puffin deletion-vector metadata through Substrait and map it to Velox `FileContent::kDeletionVector`.
- Validate deletion vectors per delete file while continuing to fall back for equality deletes and malformed metadata.
- Offload Iceberg V3 `WriteDeltaExec` operations for DELETE, UPDATE, and MERGE when enhanced native writes are enabled.
- Preserve Iceberg's existing driver-side commit, validation, conflict detection, and abort handling.
- Seed native writes with existing deletion vectors so repeated mutations union deleted positions and replace only the prior active DV.
- Keep Spark 3.3 and Iceberg V2 behavior unchanged.
- Cover Spark 3.4, 3.5, 4.0, and 4.1 compatibility, native writer behavior, partitioned tables, and split-boundary reads.

### Gluten version

main branch

Contributor guide

Open the contributing guide

Research direction

Start by tracing the Substrait deletion-vector metadata path and Velox's existing deletion-vector writer support, then follow Iceberg V3 WriteDeltaExec handling for DELETE, UPDATE, and MERGE. Done means native V3 deletion-vector reads and writes work across the listed Spark versions, partitioned tables, and split boundaries while equality deletes, malformed metadata, Spark 3.3, and Iceberg V2 retain their stated fallback or existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.