elastic / elastic/elasticsearch
MaxDocCount for term aggregation
- Dominant language
- Java
- Stars
- 77.9k
- Forks
- 26.1k
- PR merge metrics
- PR metrics pending
Description
This was already requested (github issues) couple of times and discussed on forums.
I had the need to implement this on my own.
Related:
- https://github.com/elastic/elasticsearch/issues/19565
- https://discuss.elastic.co/t/max-doc-count-in-terms-aggragation/131142
- https://github.com/elastic/elasticsearch/issues/21714
This feature is required in the cases when I want to apply range on buckets to be retrieved. Bucket Selector Aggregation cannot be used here when there are millions of buckets due to fact that all buckets are retrieved before applying the selector.
Facts:
- It works for me, but with some glitches I would like to fix
- This is WIP. I just started working on this and I would be happy to receive some help from anyone willing to participate in this.
Help needed to discover the way to make the results respect "requiredSize" (size) option.
Imagine there are buckets with size of 200000 all the way to zero. Currently, when I set like this (see agg below), buckets with the count greater than 10000 are taken in account but I get no results.
To get the results I need to set the "size" to some greater number (e.g 15000). But, results will be within the range. I guess that there should be some decrement (or filter on increment) applied for total buckets found.
If someone could point me to the right direction, I could implement this faster most probably. In any case I will continue with the experiment. I am satisfied with the result. Like I said, bucket selector cannot stand too many buckets but this approach looks fine (except that issue with size)
`{
"terms": {
"field": "applicant.applicant_url",
"size":1000,
"max_doc_count": 10000,
"min_doc_count": 1000
}
`
Issued PR: https://github.com/elastic/elasticsearch/pull/74752
Contributor guide
Research direction
Start with the terms aggregation entry point and the linked PR 74752, then trace how the size, max_doc_count, and min_doc_count options affect bucket selection. Done means the shown aggregation returns the expected buckets while respecting the required size without requiring an artificially larger size.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100