Verify to_date/to_timestamp data skipping still works after ReplaceExpressions
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Describe the problem
`BaseHoodieCatalystExpressionUtils` recurses into `ParseToDate`/`ParseToTimestamp` via the per-Spark-version `unapplyOrderPreservingDateParsing` hook (#19149) so that `to_date(col)`/`to_timestamp(col)` predicates can be mapped back to the source column for data skipping.
Both nodes are `RuntimeReplaceable`. Spark's first optimizer batch (`FinishAnalysis` -> `ReplaceExpressions`) rewrites them before filters are pushed down, so by the time `DataSkippingUtils.translateIntoColumnStatsIndexFilterExpr` sees a predicate from a real query, the `ParseToDate`/`ParseToTimestamp` shapes may no longer exist -- and their replacements (`GetTimestamp`/`Cast` trees) are not in the order-preserving whitelist. If so, `to_date`/`to_timestamp` data skipping is silently a no-op.
The existing green coverage is not representative: the `to_timestamp` case in `TestDataSkippingUtils` (around line 677) runs a test-local optimizer that applies only `OptimizeIn`, so the un-replaced node survives there but not in a real query plan.
### Action
Add an optimizer-realistic test: resolve `to_date(col, fmt) = lit`, run the full `FinishAnalysis` batch, then call `translateIntoColumnStatsIndexFilterExpr` and assert on the result. Either it proves the hook still fires, or it shows the whitelist needs the replaced shapes instead.
Found while reviewing #19405, which pins the hook directly with a unit test; this issue tracks the end-to-end reachability question.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.