apache / apache/gluten

[UT][VL] glutenNormalizeIds mis-normalizes Brand#NN string literals in TPC-H q19 (ExprId collision)

Open
#12,375 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

## Summary

`GlutenTPCHPlanStabilitySuite` → `tpch/q19` fails in `spark-test-spark40` CI for any PR that touches Velox backend Scala files. The failure is caused by a stale golden file combined with a known limitation in the ExprId normalizer.

## Affected check

`spark-test-spark40` (and `spark-test-spark41`)

## Root cause

`GlutenPlanStabilitySuite.glutenNormalizeIds()` uses the regex `(?(?` in the explain text — including TPC-H string literals. The `p_brand` filter in q19 uses values `Brand#11`, `Brand#12`, `Brand#13` (actual TPC-H spec data values). These appear unquoted in the explain output:

```
EqualTo(p_brand, Brand#12)
```

The normalizer incorrectly treats `#12` as an ExprId and remaps it sequentially based on encounter order. The suite code itself documents this limitation at line 67–68:

> *"Running all suites together in one JVM is recommended to avoid ExprId normalization issues where string constants (e.g., Brand#23 in TPCH q19) may collide with ExprId numbers."*

## How it manifests

The golden file was committed in #11805 (`c37fee4e5`, 2026-03-24). Over the 264 commits since then, new optimizer rules and expressions shifted the ExprId counter. `Brand#12` now normalizes to `Brand#6` and `_pre_1#14` shifts to `_pre_1#13`, causing a spurious mismatch.

Reproduced on `main` at commit `6097b59a6` (2026-06-25, [MINOR][VL] Build Arrow 18 with patch for Power #12344) without any pending PR:
```
Tests: succeeded 21, failed 1 ← tpch/q19
BUILD FAILURE
```

## Short-term fix

Refresh `q19/explain.txt` via `SPARK_GENERATE_GOLDEN_FILES=1` — tracked in #12374.

## Long-term fix

Make `glutenNormalizeIds` skip `#N` patterns that appear inside string literal contexts (i.e., where the `#` is preceded by non-whitespace word characters that are not a column/expression name). This would prevent TPC-H brand values like `Brand#12` from being incorrectly normalized.

Contributor guide

Open the contributing guide

Research direction

Start at GlutenPlanStabilitySuite.glutenNormalizeIds() and the documented limitation around lines 67–68. Reproduce the tpch/q19 failure in spark-test-spark40 or spark-test-spark41, then verify that Brand#11, Brand#12, and Brand#13 are no longer remapped while ExprId normalization still works. Run the relevant plan stability suite and confirm q19 matches its golden file.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.