apache / apache/druid

Error bounds / probabilities / skewness as first-class Druid query results

Open
#7,160 2 comments 0 reactions 0 assignees View on GitHub
Area - Query UI Area - Querying Discuss
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

Describing [Online Aggregation](https://github.com/apache/incubator-druid/issues/7087), I suggested that when Broker sends partial results back to the client it also sends a flag indicating that the partial aggregation results may be skewed. It may also send estimated error / confidence intervals of the partial aggregation values, if it is able to compute them for the given aggregation function, and if the user opts to receive such data.

I think this idea shouldn't be confined to partial query results during online aggregation and could equally apply to "final" query results (equivalent to "offline" query results).

Some of the sources of inconsistencies / error / variance:
- Limitations of the distributed query execution: see an issue regarding [TopN Aliasing](https://github.com/apache/incubator-druid/issues/1134) (where @drcrallen [gives a direct example](https://github.com/apache/incubator-druid/issues/1134#issuecomment-74913074) of variance between topN results from different data nodes. See also a related [join query discussion](https://github.com/apache/incubator-druid/issues/4040).
- Time trends for single-valued query types such as topN and groupBy: relative results of for different dimension values (grouping keys) may have a time trend that is averaged out by the final aggregation and thus invisible to the user.
- Significant variance between different partitions within the same time interval might mean that there is simply not enough data to draw reliable conclusions from the final results. In some contexts this is OK (usually when making a topN or count query we are really interested in absolute values, for example, `count(log_lines) where error=true`), but in other cases, namely when we are interested in proportion, relative values, and trends, we should at least make users aware of the fact that results may include significant error.
- Natural probabilistic nature of many Druid aggregators such as quantiles, sketches, HLL, etc. including those that back up classic SQL query types under the covers. See, for example, #6099 and the discussion on [replacing the default implementation behind DISTINCT COUNT from one probabilistic structure to another](https://github.com/apache/incubator-druid/issues/6814).
- Something else?

As well as with [Online Aggregation](https://github.com/apache/incubator-druid/issues/7087), work should be done on both the backend (Druid itself) and frontend side of UIs querying into Druid to support this and bring value to users.

In terms of [antifragility](https://en.wikipedia.org/wiki/Antifragility), the current Druid's error-oblivious approach to query results may be classified as fragile. The approach that makes errors first-class query results might be classified as resilient or perhaps even antifragile because it might help users to learn something new about their data during abrupt events.

FYI @gianm @mistercrunch @vogievetsky @julianhyde @leerho @weijietong

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked Online Aggregation issue and the TopN Aliasing, join query, and probabilistic-aggregator discussions. Map the Broker, Druid backend, aggregators, and frontend UIs involved; the work is done only when the scope and acceptance criteria for exposing skew, error bounds, probabilities, or confidence intervals as query results are agreed for both partial and final results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, data, frontend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.