opensearch-project / opensearch-project/sql
[BUG] Query with aggregations returns incorrect number of results
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 176
- Forks
- 229
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 43
Description
What is the bug?
When running the following query from the UI in workbench, it is not possible to obtain over 200 rows as a result:
select field1, count(*)
from grouping_test
where field1 is not null
group by 1
Also, wrapping the query around another select statement doesn't work either:
select count(*)
from (select field1, count(*)
from grouping_test
where field1 is not null
group by 1) t
After changing the query size limit, the number changes from 200 to 1000 instead of 10000:
How can one reproduce the bug?
Steps to reproduce the behavior:
- Run the following python code to create an index and load 3000 rows to it:
import json
import time
from opensearchpy import OpenSearch
from opensearchpy.client.cat import CatClient
from opensearchpy.client.indices import IndicesClient
def get_bulk_upload_json(data: list) -> str:
json_data = []
for item in data:
ind = {"index": {"_id": item["_id"]}}
json_data.append(json.dumps(ind))
del item["_id"]
json_data.append(
json.dumps(item, default=str)
)
json_data = "\n".join(json_data)
json_data = json_data + "\n"
return json_data
os_client = OpenSearch(
hosts=[
{'host': 'host',
'port': 443,
},
],
http_compress=True,
http_auth=('osadmin', 'password'),
use_ssl=True
)
cat_client = CatClient(os_client)
indices_client = IndicesClient(os_client)
index_name = "grouping_test"
if indices_client.exists(index_name):
indices_client.delete(index_name)
indices_client.create(
index_name,
body={
"mappings": {
"properties": {
"field1": {"type": "integer"}
},
},
},
)
rows = [{"field1": i, "_id": i} for i in range(0, 3000)]
os_client.bulk(body=get_bulk_upload_json(rows), index=index_name, wait_for_active_shards='all')
while 3000 > int(
cat_client.count(index=index_name, format="json")[0]["count"]
):
time.sleep(5)
print(cat_client.count(index=index_name, format="json")[0]["count"])
- Run the following query in workbench
select count(*)
from (select field1, count(*)
from grouping_test
where field1 is not null
group by 1) t
What is the expected behavior?
When running the query, it should return either the full set of results or the number of the query size settings. When wrapping the query in another select it should return 3000.
What is your host/environment?
AWS OpenSearch 2.7
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the nested aggregation query in Workbench against the 3,000-row grouping_test index described in the issue, then trace how aggregation results and query-size limits are handled. Done means the direct query returns the full set or configured limit, and the wrapped count query returns 3,000.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100