apache / apache/hudi

[BUG] Data skipping: left(col, n) predicates never prune -- whitelist arm matches a shape the optimizer eliminated (SPARK-38240)

Open
#19,479 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

Same bug class as #19446, different expression. `Left` has been RuntimeReplaceable in every Spark version Hudi supports (3.3 through 4.2): the optimizer's ReplaceExpressions rewrites `left(str, n)` into `Substring(str, 1, n)` before any filter reaches Hudi's file pruning. The order-preserving whitelist in `BaseHoodieCatalystExpressionUtils.OrderPreservingTransformation` only matches the pre-replacement `Left` shape, so `left(col, n) = '...'` predicates translate to TrueLiteral and data skipping silently prunes nothing.

The arm even sits under a comment citing SPARK-38240 -- the arity of a node that never arrives was kept up to date across Spark upgrades.

**Evidence**

- `javap -cp spark-catalyst_2.12-3.3.4.jar org.apache.spark.sql.catalyst.expressions.Left` -> `implements RuntimeReplaceable` (same for 3.4.3, 3.5.6, 4.0.2, 4.1.1, 4.2.0); replacement is `Substring(str, Literal(1), len)` (visible in bytecode).
- Live optimizer probe: `left(B, 4) > '2021'` optimizes to `substring(B#0, 1, 4)`, which the whitelist does not match (`lower(B)` by contrast still matches).
- Archaeology: the `Left` arm was added together with `ParseToDate`/`ParseToTimestamp` in HUDI-3594 (#4996, Mar 2022); all three were already RuntimeReplaceable in Spark 3.2, so the arm has never fired on the read path.

**Suggested fix**

Add a prefix-only Substring arm next to the existing string arms:

```scala
case Substring(OrderPreservingTransformation(attrRef), Literal(1, _), _) => Some(attrRef)
```

(a non-1 `pos` is NOT order-preserving), delete the dead `Left` arm, and add a `left(B, 4) > '...'` row to the full-optimizer method source introduced in #19474. Note prefix-substring over the string min/max stats is order-preserving and total, so it needs neither the format gate nor extra null handling.

**Environment**: master (found during review of #19474).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in BaseHoodieCatalystExpressionUtils.OrderPreservingTransformation and inspect the existing string-expression arms, especially Left and Substring. Then read the full-optimizer method source introduced in #19474 and add the reported left(B, 4) predicate case; done means the optimized predicate is recognized for prefix substring pruning and the relevant tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data-engineering, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.