apache / apache/gluten

Native-handled PushedFilters are not marked with `*` in file-source scan plans

Open
#12,753 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Description

Gluten's file-source scan transformers (`FileSourceScanExecTransformerBase` and its subclasses: `FileSourceScanExecTransformer`, `DeltaScanTransformer`, `HudiScanTransformer`) push **every** filter conjunct into the native scan via `PushDownFilterToScan` and evaluate them as exact row-level filters. As a result the paired `FilterExecTransformer` becomes a no-op (`FilterExecTransformerBase.isNoop`) and no separate post-scan `Filter` is needed.

Spark marks exactly this situation with a leading `*` on a scan's `PushedFilters` metadata — the `RowDataSourceScanExec` convention for a filter the source fully evaluates itself. Gluten, however, inherits `FileSourceScanLike`'s rendering, which leaves `PushedFilters` **unmarked**:

```
PushedFilters: [IsNotNull(id), LessThan(id,5)] <- Gluten today
PushedFilters: [*IsNotNull(id), *LessThan(id,5)] <- what the state actually is
```

So the plan output under-reports what the native scan actually does. This is purely a plan-rendering gap — the filters are already fully pushed down (present in the scan's `dataFilters` / Substrait `NativeFilters`); only the `*` marker is missing.

This also surfaces when running upstream engines' plan-string assertions against Gluten (e.g. Delta's CDF filter-pushdown tests assert the `*`-marked form).

### Expected behavior

Each fully-pushed filter entry in a `FileSourceScanExecTransformer` scan's `PushedFilters` should be rendered with a leading `*`, matching Spark's `RowDataSourceScanExec` convention. Vanilla `FileSourceScanExec` nodes (e.g. the AQE Initial Plan) must remain unmarked, since they still have a real `Filter` above them.

### Notes

The change lives in the common `gluten-substrait` module (hence `[CORE]`). The observable golden updates are Velox-only (TPC-H / TPC-DS / gluten-tpch plan-stability). Addressed by apache/gluten#12626.

Once this is fixed we should remove these tests from `.github/workflows/util/delta-spark-ut/known-failures.txt`

```
org.apache.spark.sql.delta.DeltaCDCIdColumnMappingSuite#filters with special characters in name should be pushed down - column mapping id mode
org.apache.spark.sql.delta.DeltaCDCNameColumnMappingSuite#filters with special characters in name should be pushed down - column mapping name mode
org.apache.spark.sql.delta.DeltaCDCSQLIdColumnMappingSuite#filters with special characters in name should be pushed down - column mapping id mode
org.apache.spark.sql.delta.DeltaCDCSQLNameColumnMappingSuite#filters with special characters in name should be pushed down - column mapping name mode
org.apache.spark.sql.delta.DeltaCDCSQLSuite#select individual column should push down filters
org.apache.spark.sql.delta.DeltaCDCSQLWithCatalogOwnedBatch100Suite#select individual column should push down filters
org.apache.spark.sql.delta.DeltaCDCSQLWithCatalogOwnedBatch1Suite#select individual column should push down filters
org.apache.spark.sql.delta.DeltaCDCSQLWithCatalogOwnedBatch2Suite#select individual column should push down filters
org.apache.spark.sql.delta.DeltaCDCScalaSuite#filters should be pushed down
org.apache.spark.sql.delta.DeltaCDCScalaWithCatalogOwnedBatch100Suite#filters should be pushed down
org.apache.spark.sql.delta.DeltaCDCScalaWithCatalogOwnedBatch1Suite#filters should be pushed down
org.apache.spark.sql.delta.DeltaCDCScalaWithCatalogOwnedBatch2Suite#filters should be pushed down
org.apache.spark.sql.delta.DeltaCDCScalaWithDeletionVectorsSuite#filters should be pushed down
```

Contributor guide

Open the contributing guide

Research direction

Start in the common gluten-substrait module by reading FileSourceScanExecTransformerBase and its FileSourceScanExecTransformer, DeltaScanTransformer, and HudiScanTransformer subclasses, then inspect the inherited FileSourceScanLike plan rendering. Compare the Velox TPC-H, TPC-DS, and gluten-tpch golden plans with Spark's marked output; done means transformer scans show marked filters while vanilla scans remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.