opensearch-project / opensearch-project/sql
[BUG] Simple `top` queries on high-cardinality fields are very slow
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 176
- Forks
- 229
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 43
Description
What is the bug?
Simple top queries can take more than 30s if done on fields with high cardinality (>1mil distinct values). This happens even if there's a small top limit such as the default top 10.
How can one reproduce the bug?
Steps to reproduce the behavior:
- Index data with lots of distinct
request_idvalues (>1mil,keyword-typed) source=idx | top request_id- Measure
Explain plan:
> echo 'source=logs | top request_id' | ppl --explain
= Calcite Plan =
== Logical ==
LogicalSystemLimit(fetch=[10000], type=[QUERY_SIZE_LIMIT])
LogicalProject(request_id=[$0], count=[$1])
LogicalFilter(condition=[=($2, 10)])
LogicalProject(request_id=[$0], count=[$1], _row_number_rare_top_=[ROW_NUMBER() OVER (ORDER BY $1 DESC, $0)])
LogicalAggregate(group=[{0}], count=[COUNT()])
LogicalProject(request_id=[$60])
CalciteLogicalIndexScan(table=[[OpenSearch, logs]])
== Physical ==
EnumerableLimit(fetch=[10000])
EnumerableCalc(expr#0..2=[{inputs}], expr#3=[10], expr#4=[=($t2, $t3)], proj#0..1=[{exprs}], $condition=[$t4])
EnumerableWindow(window#0=[window(order by [1 DESC, 0] rows between UNBOUNDED PRECEDING and CURRENT ROW aggs [ROW_NUMBER()])])
CalciteEnumerableIndexScan(table=[[OpenSearch, logs]], PushDownContext=[[AGGREGATION-rel#50:LogicalAggregate.NONE.[](input=RelSubset#49,group={0},count=COUNT())], OpenSearchRequestBuilder(sourceBuilder={
"aggregations": {
"composite_buckets": {
"composite": {
"size": 10000,
"sources": [
{
"request_id": {
"terms": {
"field": "request_id",
"missing_bucket": true,
"missing_order": "first",
"order": "asc"
}
}
}
]
}
}
},
"from": 0,
"size": 0,
"timeout": "1m"
}, requestedTotalSize=2147483647, pageSize=null, startFrom=0)])
The inner DSL in this case takes ~300ms (index cache disabled), while the PPL takes ~43800ms on my machine. However, it's also ordering keys lexically instead of by count using OpenSearch's own top_values aggregation.
What is the expected behavior?
At least this case of a trivial top aggregation (without trailing consuming commands) should be fast. OpenSearch itself can do this quickly (133 ms) with straightforward DSL, including returning an accurate size for the "other" bucket (sum_other_doc_count):
{
"size": 0,
"aggs": {
"top_values": {
"terms": {
"field": "request_id",
"size": 10
}
}
}
}
{
"took": 99,
"timed_out": false,
"terminated_early": true,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 10000,
"relation": "gte"
},
"max_score": null,
"hits": []
},
"aggregations": {
"top_values": {
"doc_count_error_upper_bound": 4,
"sum_other_doc_count": 999990,
"buckets": [
{
"key": "00000067-142d-4d90-8a2c-28409600adbd",
"doc_count": 1
},
{
"key": "00001910-7d05-4e6a-9716-2c74137475e6",
"doc_count": 1
},
{
"key": "00002676-4f8c-428a-a29a-b862ca672cb7",
"doc_count": 1
},
{
"key": "00003943-8463-46f2-a92b-641b4afbafb1",
"doc_count": 1
},
{
"key": "00003f95-b049-41de-b9be-03aae071ad99",
"doc_count": 1
},
{
"key": "00005502-587d-4b00-865c-86f2af5502c2",
"doc_count": 1
},
{
"key": "00005754-ad22-4894-a9b5-dba3007ff19a",
"doc_count": 1
},
{
"key": "00005c50-97d3-4c11-ad74-90a3219ca815",
"doc_count": 1
},
{
"key": "000066c7-0301-45aa-98b1-d4997a590214",
"doc_count": 1
},
{
"key": "00007501-890e-4f0b-a688-2653ade97f8b",
"doc_count": 1
}
]
}
}
}
What is your host/environment?
- mainline, also tested 3.5
- linux (AL2023), jdk 25
Do you have any screenshots?
N/A
Do you have any additional context?
Seems to be that we don't have any top_values aggregation pushdown at all, so this query turns into a coordinator scan.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The reported entry point is a PPL top query; start with the ppl --explain logical and physical plans and their aggregation pushdown, especially the generated composite_buckets request. Compare it with the shown terms/top_values request; done means trivial top queries avoid the coordinator scan and return the expected top results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- backend-api-design, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100