opensearch-project / opensearch-project/sql

[BUG] `doc_count` values for a terms/multi-terms aggregation is approximate

Open
#4,833 3 comments 0 reactions 1 assignee View on GitHub

@LantaoJin is already working on this.

Since Nov 20, 2025.

aggregation bug clickbench dependencies PPL
Dominant language
Java
Stars
176
Forks
229
Avg merge
2d 21h
Merged PRs (30d)
43

Description

What is the bug?
The correctness issue happens on following PPL queries on Clickbench:
Q15 and its plan
Q16 and its plan
Q17 and its plan
Q22 and its plan
Q34 and its plan
Q35 and its plan

For example, the query of Q16 is fully pushed down to DSL.
The following DSL query returns incorrect (approximate) TopK:

{
  "from": 0,
  "size": 0,
  "timeout": "1m",
  "aggregations": {
    "UserID": {
      "terms": {
        "field": "UserID",
        "size": 10,
        "min_doc_count": 1,
        "shard_min_doc_count": 0,
        "show_term_doc_count_error": false,
        "order": [
          {
            "count()": "desc"
          },
          {
            "_key": "asc"
          }
        ]
      },
      "aggregations": {
        "count()": {
          "value_count": {
            "field": "_index"
          }
        }
      }
    }
  }
}

The result is

{"took":4205,"timed_out":false,"terminated_early":true,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":10000,"relation":"gte"},"max_score":null,"hits":[]},"aggregations":{"UserID":{"doc_count_error_upper_bound":-1,"sum_other_doc_count":99890741,"buckets":[{"key":1313338681122956954,"doc_count":29097,"count()":{"value":29097}},{"key":1907779576417363396,"doc_count":25331,"count()":{"value":25331}},{"key":2305303682471783379,"doc_count":10594,"count()":{"value":10594}},{"key":7982623143712728547,"doc_count":7584,"count()":{"value":7584}},{"key":6018350421959114808,"doc_count":6678,"count()":{"value":6678}},{"key":1090981537032625727,"doc_count":6197,"count()":{"value":6197}},{"key":5730251990344211405,"doc_count":6019,"count()":{"value":6019}},{"key":7280399273658728997,"doc_count":6018,"count()":{"value":6018}},{"key":835157184735512989,"doc_count":5211,"count()":{"value":5211}},{"key":770542365400669095,"doc_count":4027,"count()":{"value":4027}}]}}}

But the expected result is

UserID,count(1)
1313338681122956954,29097
1907779576417363396,25333
2305303682471783379,10597
7982623143712728547,7584
6018350421959114808,6678
7280399273658728997,6411
1090981537032625727,6197
5730251990344211405,6019
835157184735512989,5211
770542365400669095,4906

The actual doc_count is 6411 by querying following DSL query:

{
  "size": 0,
  "track_total_hits": true,
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "UserID": "7280399273658728997"
          }
        }
      ]
    }
  }
}

This is an OpenSearch Core issue, cannot be fixed within SQL plugin.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.