Graylog2 / Graylog2/graylog2-server

Consider using composite aggregation in AggregationEventProcessor

Open
#16,131 0 comments 0 reactions 0 assignees View on GitHub
alerting feature improvement performance triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

The AggregationEventProcessor has the following TODO:

https://github.com/Graylog2/graylog2-server/blob/master/graylog2-server/src/main/java/org/graylog/events/processor/aggregation/PivotAggregationSearch.java#L456-L470

```
// The pivot search type (as of Graylog 3.1.0) is using the "terms" aggregation under
// the hood. The "terms" aggregation is meant to return the "top" terms and does not allow
// and efficient retrieval and pagination over all terms.
// Using Integer.MAX_VALUE as a limit can be very expensive with high cardinality grouping.
// The ES documentation recommends to use the "Composite" aggregation instead.
//
// See the ES documentation for more details:
// https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-size
// https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-composite-aggregation.html
//
// The "Composite" aggregation is only available since ES version 6.1, unfortunately.
//
// TODO: Either find a way to use the composite aggregation when the ES version in use is
// recent enough, and/or use a more conservative limit here and make it configurable
// by the user.
```

Since we do not support ES 6.0 or older anymore, we could change that.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.