opensearch-project / opensearch-project/OpenSearch
Add FIRST/LAST support to DSL - for sorting aggregation results by a keyword field
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 13.7k
- Forks
- 3k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 108
Description
I'm performing a term aggregation, and I want to sort my results based on a keyword field (fileInfo.fileName).
While sorting works great for numeric/date columns using MIN/MAX function, there is no way to do the same for a non-numeric field.
For my dataset, I know that the field 'fileInfo.fileName' is identical for all items in each bucket. Therefore, any function returning any of the file names would be sufficient.
My query:
POST fileinstances/_search
{
"size": 0,
"aggregations": {
"types": {
"terms": {
"field": "fileInfo.fileQualifier",
"size": 250,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"firstTimestamp": "desc"
}
]
},
"aggregations": {
"firstTimestamp": {
"min": {
"field": "@timestamp"
}
},
"file_name": {
"terms": {
"field": "fileInfo.fileName.keyword"
}
},
"agents_count": {
"cardinality": {
"field": "agentId.keyword"
}
},
"first_file": {
"top_hits": {
"from": 0,
"size": 1,
"_source": {
"includes": [
"fileInfo.fileQualifier",
"fileInfo.fileName",
"fileInfo.fileDescription",
"fileInfo.hash",
"fileInfo.publisher",
"fileInfo.fileSize"
],
"excludes": []
},
"sort": [
{
"@timestamp": {
"order": "asc"
}
}
]
}
}
}
},
"filtered_count": {
"cardinality": {
"field": "fileInfo.fileQualifier"
}
}
}
}
Any trial I did for ordering by fileName was not successful, since the search engine doesn't support order by pipeline aggregations.
It ended with error like this:
"error" : {
"root_cause" : [
{
"type" : "aggregation_execution_exception",
"reason" : "Invalid aggregation order path [file_name]. Buckets can only be sorted on a sub-aggregator path that is built out of zero or more single-bucket aggregations within the path and a final single-bucket or a metrics aggregation at the path end."
}
],
Describe the solution you'd like
I'd like to be able to do:
"firstFilename": {
"min": {
"field": "fileInfo.fileName.keyword"
}
and then order the result by 'firstFilename'.
Alternatively, use another function such as FIRST/LAST instead of min/max.
Describe alternatives you've considered
Transform index might have been a solution. However, it is a lot more complicated and also doesn't support all types of aggregations like top_hits and other complex queries.
Additional context
Here is how FIRST/LAST has been implemented in SQL engine, therefore it is surely possible:
https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-functions-aggs.html#sql-functions-aggs-first
The issue is also referenced here:
https://github.com/elastic/elasticsearch/issues/37198
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source files, tests, or implementation entry points are named. Start from the aggregation query and the reported invalid aggregation order path error, then review how metric aggregations and terms-bucket ordering are represented; done means a keyword-field FIRST/LAST-style aggregation can be used to order aggregation results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100