Indicating time trends in topN results
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
This issue elaborates on one of the aspects mentioned in the discussion about [Error bounds / probabilities / skewness as first-class Druid query results](https://github.com/apache/incubator-druid/issues/7160): time trends for single-valued query types such as topN and groupBy. In different time intervals, results may be different, for example:
```
interval=00:00_01:00
country website_visits
China 2000
Argentina 1000
USA 900
```
...
```
interval=05:00_06:00
country website_visits
Argentina 1500
USA 1300
China 1000
```
-- China goes to bed, Americas wake up.
When we aggregate just those 5 hours, there is a relative trend for each dimension value (country). We may detect this when aggregating results on Broker and send a enum along with the value for each dimension value or grouping key:
- consistent upward trend
- consistent downward trend
- no trend (the relative weight of the key stays realtively stable), or a cycling trend: relative weight goes up and down and finishes approximately at the same place where it has started.
- big variance, but no particular trend.
Then in query UIs, consistent upward and downward trends may be represented as small green and red arrows (those that are used in stock market interfaces :)
Big variance, but no particular trend may be indicated by something like an asterisk or different tone of the row or an exclamation mark in a circle, indicating that the total aggregate value probably has low statistical significance, or there is some problem with the data (see #7160).
FYI @mistercrunch @julianhyde @vogievetsky
Contributor guide
Research direction
Begin by reading the linked discussion in issue #7160 and the examples for topN and groupBy results. The work would need a defined approach for detecting relative trends during Broker aggregation and representing those results for query UIs, with agreement on what each trend classification means.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- analytics, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100