[VL] Support Iceberg V3 deletion vectors in Velox
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Description
Add native Iceberg format-version 3 deletion-vector support to the Velox backend.
Today Gluten falls back when reading Puffin deletion vectors and row-level DELETE, UPDATE, and MERGE operations do not use the enhanced native Iceberg writer. Velox already contains the required deletion-vector writer support.
Proposed scope:
- Pass Puffin deletion-vector metadata through Substrait and map it to Velox `FileContent::kDeletionVector`.
- Validate deletion vectors per delete file while continuing to fall back for equality deletes and malformed metadata.
- Offload Iceberg V3 `WriteDeltaExec` operations for DELETE, UPDATE, and MERGE when enhanced native writes are enabled.
- Preserve Iceberg's existing driver-side commit, validation, conflict detection, and abort handling.
- Seed native writes with existing deletion vectors so repeated mutations union deleted positions and replace only the prior active DV.
- Keep Spark 3.3 and Iceberg V2 behavior unchanged.
- Cover Spark 3.4, 3.5, 4.0, and 4.1 compatibility, native writer behavior, partitioned tables, and split-boundary reads.
### Gluten version
main branch
Contributor guide
Research direction
Start by tracing the Substrait deletion-vector metadata path and Velox's existing deletion-vector writer support, then follow Iceberg V3 WriteDeltaExec handling for DELETE, UPDATE, and MERGE. Done means native V3 deletion-vector reads and writes work across the listed Spark versions, partitioned tables, and split boundaries while equality deletes, malformed metadata, Spark 3.3, and Iceberg V2 retain their stated fallback or existing behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100