apache / apache/pinot

Time column predicates are invalidating use of star-tree indices

Open
#6,478 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

In some cases, time-column predicates are disqualifying queries from using star-trees, when they would otherwise be used. Specifically, this appears to be the case when using a dynamic timestamp comparison against the now() operator:

`SELECT dimension, SUM(metric) AS totalMetrics FROM myTable WHERE otherDimension='filterValue' AND eventTimestamp >= cast(now() - 172800000 as long) GROUP BY 1 ORDER BY 2 DESC LIMIT 10`

where a query that uses a static value for timestamp comparison will use a star-tree:
`SELECT dimension, SUM(metric) AS totalMetrics FROM myTable WHERE otherDimension='filterValue' AND eventTimestamp >= 1611161288000 GROUP BY 1 ORDER BY 2 DESC LIMIT 10`

Thanks to @mayankshriv, I'll try to put together a PR to address.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the supplied queries and tracing how dynamic time-column predicates affect star-tree eligibility. Done means the now()-based comparison can use a star-tree when otherwise eligible, with coverage for both dynamic and static timestamp predicates.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.