apache / apache/hudi

Expression index should be leveraged only when the predicate matches the expression of interest

Open
#17,479 0 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Bug Description

**What happened:**
say we create an expression index as below
```
create index idx_datestr on tableName using column_stats(ts) options(expr='from_unixtime', format='yyyy-MM-dd HH:mm')
```

And the expectation is that, if a query with predicate "where from_unixtime(ts, 'yyyy-MM-dd') = '1970-01-01'" is supplied, expression index will be used.
Eg query:
```
select id, name from tableName where from_unixtime(ts, 'yyyy-MM-dd') = '1970-01-01'
```

But if the query contains the data column directly, we should not be looking up in expression index for pruning, but such query are looking up in expression index and hits exception due to casting issue. But we swallow the failure silently and move on to next index.

Image

**What you expected:**
Only when the expression matches, we should lookup in expression index, if not, we should fallback to other indices available.

**Steps to reproduce:**
1.
2.
3.

### Environment

**Hudi version:**
**Query engine:** (Spark/Flink/Trino etc)
**Relevant configs:**

### Logs and Stack Trace

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing Hudi's expression-index lookup during predicate pruning for the shown from_unixtime query and compare it with a predicate using ts directly. Reproduce with the CREATE INDEX and SELECT examples; done means the expression index is consulted only when the predicate expression matches, while other indices remain available otherwise.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.