Expression index should be leveraged only when the predicate matches the expression of interest
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Bug Description
**What happened:**
say we create an expression index as below
```
create index idx_datestr on tableName using column_stats(ts) options(expr='from_unixtime', format='yyyy-MM-dd HH:mm')
```
And the expectation is that, if a query with predicate "where from_unixtime(ts, 'yyyy-MM-dd') = '1970-01-01'" is supplied, expression index will be used.
Eg query:
```
select id, name from tableName where from_unixtime(ts, 'yyyy-MM-dd') = '1970-01-01'
```
But if the query contains the data column directly, we should not be looking up in expression index for pruning, but such query are looking up in expression index and hits exception due to casting issue. But we swallow the failure silently and move on to next index.
**What you expected:**
Only when the expression matches, we should lookup in expression index, if not, we should fallback to other indices available.
**Steps to reproduce:**
1.
2.
3.
### Environment
**Hudi version:**
**Query engine:** (Spark/Flink/Trino etc)
**Relevant configs:**
### Logs and Stack Trace
_No response_
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing Hudi's expression-index lookup during predicate pruning for the shown from_unixtime query and compare it with a predicate using ts directly. Reproduce with the CREATE INDEX and SELECT examples; done means the expression index is consulted only when the predicate expression matches, while other indices remain available otherwise.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100