[VL][DELTA] Support reading Delta Lake Change Data Feed (CDF) without falling back to vanilla Spark
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Description
## Description
Gluten currently does not offload reads of Delta tables' **Change Data Feed** (`spark.read.format("delta").option("readChangeFeed", "true")...` or the `table_changes()` SQL function). These queries run entirely on vanilla Spark instead of the Velox backend.
## Why it falls back today
A normal Delta scan is a `FileSourceScanExec` whose `relation.fileFormat` is a `DeltaParquetFileFormat`. Gluten's `OffloadDeltaScan` only matches that exact case and rewrites it into a `DeltaScanTransformer`:
```scala
case scan: FileSourceScanExec
if scan.relation.fileFormat.getClass == classOf[DeltaParquetFileFormat] =>
DeltaScanTransformer(scan)
```
CDF reads do **not** produce that plan. Delta builds them through `CDCReader.DeltaCDFRelation`, a generic `BaseRelation` whose `buildScan` returns RDD[Row]
Because the resulting plan is not a `FileSourceScanExec` over `DeltaParquetFileFormat`, `OffloadDeltaScan` never matches it, so the entire query (scan + projections building the metadata columns) stays on vanilla Spark.
## Proposed work
- Recognize the CDF scan path (`DeltaCDFRelation` / the CDC file indexes) and offload the underlying parquet reads to Velox.
- Materialize the synthesized `_change_type` / `_commit_version` / `_commit_timestamp` columns (literals + projections) so they can be produced natively rather than forcing a fallback.
- Add `gluten-ut` coverage for batch CDF reads (`readChangeFeed` and `table_changes()`), including add/remove/cdc-file combinations and column mapping.
### Gluten version
main branch
Contributor guide
Research direction
Start with OffloadDeltaScan and trace how DeltaCDFRelation, CDCReader, and the CDC file indexes build the scan plan. Review the gluten-ut coverage needed for readChangeFeed and table_changes(), including add/remove/cdc-file combinations and column mapping. Done means batch CDF reads are offloaded to Velox with the synthesized change metadata columns produced natively.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100