opensearch-project / opensearch-project/sql

[BUG] Query with aggregations returns incorrect number of results

Open
#1,906 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug SQL
Dominant language
Java
Stars
176
Forks
229
Avg merge
2d 21h
Merged PRs (30d)
43

Description

What is the bug?

When running the following query from the UI in workbench, it is not possible to obtain over 200 rows as a result:

select field1, count(*)
from grouping_test
where field1 is not null
group by 1

Also, wrapping the query around another select statement doesn't work either:

select count(*)
from (select field1, count(*)
from grouping_test
where field1 is not null
group by 1) t
Screenshot 2023-07-26 at 7 38 58 PM

After changing the query size limit, the number changes from 200 to 1000 instead of 10000:

Screenshot 2023-07-26 at 7 43 40 PM Screenshot 2023-07-26 at 7 45 03 PM

How can one reproduce the bug?
Steps to reproduce the behavior:

  1. Run the following python code to create an index and load 3000 rows to it:
import json
import time

from opensearchpy import OpenSearch
from opensearchpy.client.cat import CatClient
from opensearchpy.client.indices import IndicesClient


def get_bulk_upload_json(data: list) -> str:
    json_data = []

    for item in data:
        ind = {"index": {"_id": item["_id"]}}
        json_data.append(json.dumps(ind))
        del item["_id"]
        json_data.append(
            json.dumps(item, default=str)
        )
    json_data = "\n".join(json_data)
    json_data = json_data + "\n"

    return json_data


os_client = OpenSearch(
    hosts=[
        {'host': 'host',
         'port': 443,
         },
    ],
    http_compress=True,
    http_auth=('osadmin', 'password'),
    use_ssl=True
)

cat_client = CatClient(os_client)
indices_client = IndicesClient(os_client)
index_name = "grouping_test"

if indices_client.exists(index_name):
    indices_client.delete(index_name)

indices_client.create(
    index_name,
    body={
        "mappings": {
            "properties": {
                "field1": {"type": "integer"}
            },
        },
    },
)

rows = [{"field1": i, "_id": i} for i in range(0, 3000)]
os_client.bulk(body=get_bulk_upload_json(rows), index=index_name, wait_for_active_shards='all')

while 3000 > int(
        cat_client.count(index=index_name, format="json")[0]["count"]
):
    time.sleep(5)

print(cat_client.count(index=index_name, format="json")[0]["count"])
  1. Run the following query in workbench
select count(*)
from (select field1, count(*)
from grouping_test
where field1 is not null
group by 1) t

What is the expected behavior?
When running the query, it should return either the full set of results or the number of the query size settings. When wrapping the query in another select it should return 3000.

What is your host/environment?
AWS OpenSearch 2.7

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the nested aggregation query in Workbench against the 3,000-row grouping_test index described in the issue, then trace how aggregation results and query-size limits are handled. Done means the direct query returns the full set or configured limit, and the wrapped count query returns 3,000.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.