apache / apache/gluten

[VL][Delta] Delta Lake MoR (Merge-on-Read) Native Support – Velox Backend

Open
#11,901 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Overview

Delta Lake MoR support in Gluten is currently partial and fragmented across read and DML paths.
This issue tracks the work required to make MoR execution **stable, native, and production-ready** on the Velox backend.

The focus is on:
- deletion vector (DV) semantics
- reducing fallback
- improving performance and correctness

This tracker is intended to organize the work and align with [Delta Lake’s MoR design](https://docs.google.com/document/d/1lv35ZPfioopBbzQ7zT82LOev7qV7x4YNLkMr2-L5E_M/edit?tab=t.0#heading=h.z89r7ifgftsi), deletion vector protocol, and Gluten’s lakehouse integration.

---

### Related Issues

- [VL] Unified design for data lake read support in Gluten + Velox #3378
- [VL] Delta Lake read enhancements #10377
- [VL] Delta Lake write support #10215
- Gluten 2026 Roadmap #11827

---

### Current Status

Progress so far:

- Delta DV / MoR read foundation has been prototyped in Gluten + Velox
- Native DV scan/read path is partially working
- Fallback still exists in some control-plane and non-hot-path operations
- Native `DELETE` path has been explored
- `UPDATE` / `MERGE` are not yet fully native

---

### Scope

This issue tracks Delta MoR work on the Velox backend, including:

- Native reads for Delta tables with deletion vectors
- Native DML paths that generate or update deletion vectors
- Correct handling of DV protocol semantics
- Performance improvements and fallback reduction

---

### Out of Scope

- Generic Delta CoW improvements (unless required for MoR)
- Non-Velox backend work
- Unrelated lakehouse features

---

### Constraints / Non-Goals

- No redesign of Delta protocol semantics
- Must remain compatible with Spark planning and control plane
- Focus is on native execution in Velox, not Spark-side refactors

---

### Priority

#### P0 – Core Read Correctness
- Native DV read correctness
- Native MoR read execution with minimal or zero fallback
- Stable build and runtime behavior in clean environments

#### P1 – Initial Native DML + Stability
- Native `DELETE` support for MoR
- Correct handling of files with existing deletion vectors
- Reduction of control-plane overhead and fallback
(e.g., Delta helper queries, JSON/log handling, histogram aggregation)

#### P2 – Full DML + Performance
- Native `UPDATE` support
- Native `MERGE` support
- Broader MoR performance optimization and workload coverage

---

### Work Areas

#### 1. MoR Read Path
- Complete native DV scan/read support
- Integrate MoR reads cleanly into Gluten + Velox planning/execution
- Reduce fallback on supported MoR queries

#### 2. MoR Write Path
- Add native `DELETE` support
- Add native `UPDATE` support
- Add native `MERGE` support
- Correctly handle rewriting/replacing DV states

#### 3. Delta Protocol Alignment
- Align with Delta deletion vector protocol semantics
- Handle DV descriptors (`u`, `p`, `i`) correctly
- Ensure correctness of:
- offsets
- size
- checksum
- cardinality
- Correct reconciliation for `(path, deletionVector.uniqueId)`

#### 4. Performance
- Improve MoR read performance
- Improve MoR write performance
- Benchmark against vanilla Spark on representative workloads
- Target equal or better performance vs Spark

#### 5. Testing and Validation
- Add unit and integration tests for MoR read/write paths
- Add regression tests for DV protocol edge cases
- Validate across partitioned and non-partitioned tables

---

### Success Criteria

- Supported MoR read queries execute natively with near-zero fallback
- Native DV read results match vanilla Spark / Delta semantics
- `DELETE` path is stable and mostly native
- MoR workloads show competitive or improved performance vs Spark

Contributor guide

Open the contributing guide

Research direction

No implementation files or tests are named. Start by reading the linked Delta Lake MoR design and the related Gluten issues, then map the Velox read and DML paths against the stated priorities. Done means native DV reads and stable DELETE behavior with correct protocol semantics, tests, and reduced fallback.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.