[BUG] Data skipping: left(col, n) predicates never prune -- whitelist arm matches a shape the optimizer eliminated (SPARK-38240)
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
Same bug class as #19446, different expression. `Left` has been RuntimeReplaceable in every Spark version Hudi supports (3.3 through 4.2): the optimizer's ReplaceExpressions rewrites `left(str, n)` into `Substring(str, 1, n)` before any filter reaches Hudi's file pruning. The order-preserving whitelist in `BaseHoodieCatalystExpressionUtils.OrderPreservingTransformation` only matches the pre-replacement `Left` shape, so `left(col, n) = '...'` predicates translate to TrueLiteral and data skipping silently prunes nothing.
The arm even sits under a comment citing SPARK-38240 -- the arity of a node that never arrives was kept up to date across Spark upgrades.
**Evidence**
- `javap -cp spark-catalyst_2.12-3.3.4.jar org.apache.spark.sql.catalyst.expressions.Left` -> `implements RuntimeReplaceable` (same for 3.4.3, 3.5.6, 4.0.2, 4.1.1, 4.2.0); replacement is `Substring(str, Literal(1), len)` (visible in bytecode).
- Live optimizer probe: `left(B, 4) > '2021'` optimizes to `substring(B#0, 1, 4)`, which the whitelist does not match (`lower(B)` by contrast still matches).
- Archaeology: the `Left` arm was added together with `ParseToDate`/`ParseToTimestamp` in HUDI-3594 (#4996, Mar 2022); all three were already RuntimeReplaceable in Spark 3.2, so the arm has never fired on the read path.
**Suggested fix**
Add a prefix-only Substring arm next to the existing string arms:
```scala
case Substring(OrderPreservingTransformation(attrRef), Literal(1, _), _) => Some(attrRef)
```
(a non-1 `pos` is NOT order-preserving), delete the dead `Left` arm, and add a `left(B, 4) > '...'` row to the full-optimizer method source introduced in #19474. Note prefix-substring over the string min/max stats is order-preserving and total, so it needs neither the format gate nor extra null handling.
**Environment**: master (found during review of #19474).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in BaseHoodieCatalystExpressionUtils.OrderPreservingTransformation and inspect the existing string-expression arms, especially Left and Substring. Then read the full-optimizer method source introduced in #19474 and add the reported left(B, 4) predicate case; done means the optimized predicate is recognized for prefix substring pruning and the relevant tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100