apache / apache/hudi

Verify to_date/to_timestamp data skipping still works after ReplaceExpressions

Open
#19,446 0 comments 0 reactions 1 assignee Claimed by @voonhous View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Describe the problem

`BaseHoodieCatalystExpressionUtils` recurses into `ParseToDate`/`ParseToTimestamp` via the per-Spark-version `unapplyOrderPreservingDateParsing` hook (#19149) so that `to_date(col)`/`to_timestamp(col)` predicates can be mapped back to the source column for data skipping.

Both nodes are `RuntimeReplaceable`. Spark's first optimizer batch (`FinishAnalysis` -> `ReplaceExpressions`) rewrites them before filters are pushed down, so by the time `DataSkippingUtils.translateIntoColumnStatsIndexFilterExpr` sees a predicate from a real query, the `ParseToDate`/`ParseToTimestamp` shapes may no longer exist -- and their replacements (`GetTimestamp`/`Cast` trees) are not in the order-preserving whitelist. If so, `to_date`/`to_timestamp` data skipping is silently a no-op.

The existing green coverage is not representative: the `to_timestamp` case in `TestDataSkippingUtils` (around line 677) runs a test-local optimizer that applies only `OptimizeIn`, so the un-replaced node survives there but not in a real query plan.

### Action

Add an optimizer-realistic test: resolve `to_date(col, fmt) = lit`, run the full `FinishAnalysis` batch, then call `translateIntoColumnStatsIndexFilterExpr` and assert on the result. Either it proves the hook still fires, or it shows the whitelist needs the replaced shapes instead.

Found while reviewing #19405, which pins the hook directly with a unit test; this issue tracks the end-to-end reachability question.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.