apache / apache/druid

Explain the processing buffer sizing in documentation

Open
#8,354 0 comments 1 reaction 0 assignees View on GitHub
Area - Documentation Area - Operations
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

Currently, [the documentation just says](https://github.com/apache/incubator-druid/blob/d00747774208dbbfcb272ee7d1c30cf879887838/docs/operations/basic-cluster-tuning.md) "`druid.processing.buffer.sizeBytes` can be set to 500MB." and "A size between 500MB and 1GB is a reasonable choice for general use." (BTW, this information is repeated three times on the same page; probably we also want to restructure it somehow.)

This guideline lacks an explanation of why 500MB is desirable and when somebody would want to configure even larger buffers. For example, if somebody doesn't run groupBy queries at all, 500MB maybe be much more than ever needed for other types of queries like topN and timeseries, *unless* they use large complex aggregators (like histograms).

So the doc should explain the relation between the cardinalities of the columns involved into the query, the aggregation size, and the recommended processing buffer size, as well as what will happen if the processing buffers will be undersized or oversized.

Contributor guide

Open the contributing guide

Research direction

Start with docs/operations/basic-cluster-tuning.md and locate the three repeated explanations of druid.processing.buffer.sizeBytes. Review the existing guidance for groupBy, topN, timeseries, and complex aggregators, then document how column cardinality and aggregation size affect buffer needs, including the consequences of undersizing or oversizing. Done means the guidance is consolidated and explains when values above or below the current recommendation are appropriate.

Written by the indexing model from the issue text.

Assessment

Domain
documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.