apache / apache/druid

Calcite execution planner misses timewindow filtering

Open
#19,164 1 comment 0 reactions 0 assignees View on GitHub
Uncategorized problem report
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

Please provide a detailed title (e.g. "Broker crashes when using TopN query with Bound filter" instead of just "Broker crashes").

### Affected Version
34
The Druid version where the problem was encountered.
34
### Description
We have a production query using `IN LIST`, the query is extracting data for a given time window when being executed, the main issue is that the timefilter is being lost when execution planner parses the sql query and transfers it to other nodes.

First we have a hot tier with latest 3 months and a cold tier with 3 months to 2 years segments.
It happens that we have many queries inside the cold tier for a datasource which barely has any time-window query specified for more than 3 months.

So we are using on the service side a spring java project which transfers sql statements to druid.
I inspected historical cold nodes and seen native queries, asking claude to share similar sql query i got the following

Query 1 (88% of traffic — 47,363 hits)

SELECT pbx,
APPROX_COUNT_DISTINCT_DS_HLL(j.callId) AS a0
FROM source_v1
INNERJOIN inline_data j ON c.callId = j.d0
WHERE c.addrType = 'xxxxa'
AND c.addr = 'xxxx'
GROUPBY pbx

on the service side we are building following which return correct data as broker still does the filtering but on cold tier side, I don't see the __time filtering.

SELECT COUNT(DISTINCT callId), pbx
* FROM source_v1
* WHERE ( pbx = :pbx AND addrType = 'xxx' AND addr IN (:LIST)
* AND callId NOT IN ('id1', 'id2', ...) AND __time >= ? AND __time <= ? )
* GROUP BY pbx

Contributor guide

Open the contributing guide

Research direction

The report compares the broker SQL with native queries observed on historical cold nodes; start by tracing the Calcite execution-planner path for the shown IN LIST and time-window query. Done means the time bounds remain present in the queries transferred to cold-tier nodes, with behavior checked against the reported production pattern.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.