apache / apache/gluten

[VL][DELTA] Support reading Delta Lake Change Data Feed (CDF) without falling back to vanilla Spark

Open
#12,195 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Description

## Description

Gluten currently does not offload reads of Delta tables' **Change Data Feed** (`spark.read.format("delta").option("readChangeFeed", "true")...` or the `table_changes()` SQL function). These queries run entirely on vanilla Spark instead of the Velox backend.

## Why it falls back today

A normal Delta scan is a `FileSourceScanExec` whose `relation.fileFormat` is a `DeltaParquetFileFormat`. Gluten's `OffloadDeltaScan` only matches that exact case and rewrites it into a `DeltaScanTransformer`:

```scala
case scan: FileSourceScanExec
if scan.relation.fileFormat.getClass == classOf[DeltaParquetFileFormat] =>
DeltaScanTransformer(scan)
```

CDF reads do **not** produce that plan. Delta builds them through `CDCReader.DeltaCDFRelation`, a generic `BaseRelation` whose `buildScan` returns RDD[Row]

Because the resulting plan is not a `FileSourceScanExec` over `DeltaParquetFileFormat`, `OffloadDeltaScan` never matches it, so the entire query (scan + projections building the metadata columns) stays on vanilla Spark.

## Proposed work

- Recognize the CDF scan path (`DeltaCDFRelation` / the CDC file indexes) and offload the underlying parquet reads to Velox.
- Materialize the synthesized `_change_type` / `_commit_version` / `_commit_timestamp` columns (literals + projections) so they can be produced natively rather than forcing a fallback.
- Add `gluten-ut` coverage for batch CDF reads (`readChangeFeed` and `table_changes()`), including add/remove/cdc-file combinations and column mapping.

### Gluten version

main branch

Contributor guide

Open the contributing guide

Research direction

Start with OffloadDeltaScan and trace how DeltaCDFRelation, CDCReader, and the CDC file indexes build the scan plan. Review the gluten-ut coverage needed for readChangeFeed and table_changes(), including add/remove/cdc-file combinations and column mapping. Done means batch CDF reads are offloaded to Velox with the synthesized change metadata columns produced natively.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.