Graylog2 / Graylog2/graylog2-server
Consider using composite aggregation in AggregationEventProcessor
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
The AggregationEventProcessor has the following TODO:
https://github.com/Graylog2/graylog2-server/blob/master/graylog2-server/src/main/java/org/graylog/events/processor/aggregation/PivotAggregationSearch.java#L456-L470
```
// The pivot search type (as of Graylog 3.1.0) is using the "terms" aggregation under
// the hood. The "terms" aggregation is meant to return the "top" terms and does not allow
// and efficient retrieval and pagination over all terms.
// Using Integer.MAX_VALUE as a limit can be very expensive with high cardinality grouping.
// The ES documentation recommends to use the "Composite" aggregation instead.
//
// See the ES documentation for more details:
// https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-size
// https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-composite-aggregation.html
//
// The "Composite" aggregation is only available since ES version 6.1, unfortunately.
//
// TODO: Either find a way to use the composite aggregation when the ES version in use is
// recent enough, and/or use a more conservative limit here and make it configurable
// by the user.
```
Since we do not support ES 6.0 or older anymore, we could change that.
Contributor guide
Assessment
This issue has not been assessed yet.