Explain the processing buffer sizing in documentation
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
Currently, [the documentation just says](https://github.com/apache/incubator-druid/blob/d00747774208dbbfcb272ee7d1c30cf879887838/docs/operations/basic-cluster-tuning.md) "`druid.processing.buffer.sizeBytes` can be set to 500MB." and "A size between 500MB and 1GB is a reasonable choice for general use." (BTW, this information is repeated three times on the same page; probably we also want to restructure it somehow.)
This guideline lacks an explanation of why 500MB is desirable and when somebody would want to configure even larger buffers. For example, if somebody doesn't run groupBy queries at all, 500MB maybe be much more than ever needed for other types of queries like topN and timeseries, *unless* they use large complex aggregators (like histograms).
So the doc should explain the relation between the cardinalities of the columns involved into the query, the aggregation size, and the recommended processing buffer size, as well as what will happen if the processing buffers will be undersized or oversized.
Contributor guide
Research direction
Start with docs/operations/basic-cluster-tuning.md and locate the three repeated explanations of druid.processing.buffer.sizeBytes. Review the existing guidance for groupBy, topN, timeseries, and complex aggregators, then document how column cardinality and aggregation size affect buffer needs, including the consequences of undersizing or oversizing. Done means the guidance is consolidated and explains when values above or below the current recommendation are appropriate.
Written by the indexing model from the issue text.
Assessment
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100