Native-handled PushedFilters are not marked with `*` in file-source scan plans
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Description
Gluten's file-source scan transformers (`FileSourceScanExecTransformerBase` and its subclasses: `FileSourceScanExecTransformer`, `DeltaScanTransformer`, `HudiScanTransformer`) push **every** filter conjunct into the native scan via `PushDownFilterToScan` and evaluate them as exact row-level filters. As a result the paired `FilterExecTransformer` becomes a no-op (`FilterExecTransformerBase.isNoop`) and no separate post-scan `Filter` is needed.
Spark marks exactly this situation with a leading `*` on a scan's `PushedFilters` metadata — the `RowDataSourceScanExec` convention for a filter the source fully evaluates itself. Gluten, however, inherits `FileSourceScanLike`'s rendering, which leaves `PushedFilters` **unmarked**:
```
PushedFilters: [IsNotNull(id), LessThan(id,5)] <- Gluten today
PushedFilters: [*IsNotNull(id), *LessThan(id,5)] <- what the state actually is
```
So the plan output under-reports what the native scan actually does. This is purely a plan-rendering gap — the filters are already fully pushed down (present in the scan's `dataFilters` / Substrait `NativeFilters`); only the `*` marker is missing.
This also surfaces when running upstream engines' plan-string assertions against Gluten (e.g. Delta's CDF filter-pushdown tests assert the `*`-marked form).
### Expected behavior
Each fully-pushed filter entry in a `FileSourceScanExecTransformer` scan's `PushedFilters` should be rendered with a leading `*`, matching Spark's `RowDataSourceScanExec` convention. Vanilla `FileSourceScanExec` nodes (e.g. the AQE Initial Plan) must remain unmarked, since they still have a real `Filter` above them.
### Notes
The change lives in the common `gluten-substrait` module (hence `[CORE]`). The observable golden updates are Velox-only (TPC-H / TPC-DS / gluten-tpch plan-stability). Addressed by apache/gluten#12626.
Once this is fixed we should remove these tests from `.github/workflows/util/delta-spark-ut/known-failures.txt`
```
org.apache.spark.sql.delta.DeltaCDCIdColumnMappingSuite#filters with special characters in name should be pushed down - column mapping id mode
org.apache.spark.sql.delta.DeltaCDCNameColumnMappingSuite#filters with special characters in name should be pushed down - column mapping name mode
org.apache.spark.sql.delta.DeltaCDCSQLIdColumnMappingSuite#filters with special characters in name should be pushed down - column mapping id mode
org.apache.spark.sql.delta.DeltaCDCSQLNameColumnMappingSuite#filters with special characters in name should be pushed down - column mapping name mode
org.apache.spark.sql.delta.DeltaCDCSQLSuite#select individual column should push down filters
org.apache.spark.sql.delta.DeltaCDCSQLWithCatalogOwnedBatch100Suite#select individual column should push down filters
org.apache.spark.sql.delta.DeltaCDCSQLWithCatalogOwnedBatch1Suite#select individual column should push down filters
org.apache.spark.sql.delta.DeltaCDCSQLWithCatalogOwnedBatch2Suite#select individual column should push down filters
org.apache.spark.sql.delta.DeltaCDCScalaSuite#filters should be pushed down
org.apache.spark.sql.delta.DeltaCDCScalaWithCatalogOwnedBatch100Suite#filters should be pushed down
org.apache.spark.sql.delta.DeltaCDCScalaWithCatalogOwnedBatch1Suite#filters should be pushed down
org.apache.spark.sql.delta.DeltaCDCScalaWithCatalogOwnedBatch2Suite#filters should be pushed down
org.apache.spark.sql.delta.DeltaCDCScalaWithDeletionVectorsSuite#filters should be pushed down
```
Contributor guide
Research direction
Start in the common gluten-substrait module by reading FileSourceScanExecTransformerBase and its FileSourceScanExecTransformer, DeltaScanTransformer, and HudiScanTransformer subclasses, then inspect the inherited FileSourceScanLike plan rendering. Compare the Velox TPC-H, TPC-DS, and gluten-tpch golden plans with Spark's marked output; done means transformer scans show marked filters while vanilla scans remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100