elastic / elastic/elasticsearch

MaxDocCount for term aggregation

Open
#74,751 5 comments 1 reaction 0 assignees View on GitHub
:Analytics/Aggregations >enhancement feedback_needed Team:Analytics
Dominant language
Java
Stars
77.9k
Forks
26.1k
PR merge metrics
PR metrics pending

Description

This was already requested (github issues) couple of times and discussed on forums.
I had the need to implement this on my own.

Related:

- https://github.com/elastic/elasticsearch/issues/19565
- https://discuss.elastic.co/t/max-doc-count-in-terms-aggragation/131142
- https://github.com/elastic/elasticsearch/issues/21714

This feature is required in the cases when I want to apply range on buckets to be retrieved. Bucket Selector Aggregation cannot be used here when there are millions of buckets due to fact that all buckets are retrieved before applying the selector.

Facts:

- It works for me, but with some glitches I would like to fix
- This is WIP. I just started working on this and I would be happy to receive some help from anyone willing to participate in this.

Help needed to discover the way to make the results respect "requiredSize" (size) option.
Imagine there are buckets with size of 200000 all the way to zero. Currently, when I set like this (see agg below), buckets with the count greater than 10000 are taken in account but I get no results.
To get the results I need to set the "size" to some greater number (e.g 15000). But, results will be within the range. I guess that there should be some decrement (or filter on increment) applied for total buckets found.

If someone could point me to the right direction, I could implement this faster most probably. In any case I will continue with the experiment. I am satisfied with the result. Like I said, bucket selector cannot stand too many buckets but this approach looks fine (except that issue with size)

`{
"terms": {
"field": "applicant.applicant_url",
"size":1000,
"max_doc_count": 10000,
"min_doc_count": 1000
}
`

Issued PR: https://github.com/elastic/elasticsearch/pull/74752

Contributor guide

Open the contributing guide

Research direction

Start with the terms aggregation entry point and the linked PR 74752, then trace how the size, max_doc_count, and min_doc_count options affect bucket selection. Done means the shown aggregation returns the expected buckets while respecting the required size without requiring an artificially larger size.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.