Inconsistent results if running a limit query with a group by on a numeric dimensional field and not adding that field to order by statement post upgrade of Druid from 20.1 to .21.1
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
Please provide a detailed title (e.g. "Broker crashes when using TopN query with Bound filter" instead of just "Broker crashes").
### Affected Version
This happens during an upgrade process and we have so far seen this happen when upgrading to .21.1 and when downgrade and upgrading to .20.2
### Description
After we upgraded our cluster from .20.1 to .21.1, we noticed an issue where if we were running a query with a limit, that the aggregates values were coming back with inconsistent values. We would run the exact same queries back to back and each time they would return with wildly different values (they could be differing by factor of 100s). What we observed:
- If limit was removed from the query (including turning off auto limit) the result would be consistent
- If we have limit in place but the group by is on values that are characters/strings the result would be consistent
- If we have limit in place but the group by is on numeric values, as long as if there is an order by statement and in the order by statement we include the numeric field that is in the group by the result would be consistent
- If we do a fresh install in an environment that hasn't had druid installed before or has been wiped clean we were not able to reproduce this issue
- If we have an environment where the is a mismatch of versions in the historical nodes we see this issue. Ex: We spin up a cluster where one historical node is on version .21.1 and one is on .20.2
- When we were testing upgrading process again, we see this issue on our on premises environment, but were not able to reproduce in EC2 environment.
- We have ensure that on the environments where we are seeing this issue that all the services are on version .21.1 using the status endpoint.
Before filing this ticket, I asked around in the druid forum and I was suggested to bring the discussion here. Here is the druid forum thread: https://www.druidforum.org/t/querying-with-a-limit-and-grouping-by-a-numeric-field-returns-inconsistent-results/6680/7
Contributor guide
Research direction
Start by reproducing the reported limit and group-by query across Druid 20.2 and 21.1 Historical nodes, comparing numeric and string dimensions and the presence of an order-by clause. Review the linked Druid Forum discussion and verify versions through the status endpoint. Done means the inconsistent aggregate results are isolated and a deterministic expected result or next diagnostic is established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100