opensearch-project / opensearch-project/OpenSearch

Add FIRST/LAST support to DSL - for sorting aggregation results by a keyword field

Open
#899 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement Search:Aggregations
Dominant language
Java
Stars
13.7k
Forks
3k
Avg merge
2d 23h
Merged PRs (30d)
108

Description

I'm performing a term aggregation, and I want to sort my results based on a keyword field (fileInfo.fileName).
While sorting works great for numeric/date columns using MIN/MAX function, there is no way to do the same for a non-numeric field.
For my dataset, I know that the field 'fileInfo.fileName' is identical for all items in each bucket. Therefore, any function returning any of the file names would be sufficient.

My query:

POST fileinstances/_search
{
  "size": 0,
  "aggregations": {
    "types": {
      "terms": {
        "field": "fileInfo.fileQualifier",
        "size": 250,
        "min_doc_count": 1,
        "shard_min_doc_count": 0,
        "show_term_doc_count_error": false,
        "order": [
          {
            "firstTimestamp": "desc"
          }
        ]
      },
      "aggregations": {
        
        "firstTimestamp": {
          "min": {
            "field": "@timestamp"
          }
        },
        "file_name": {
          "terms": {
            "field": "fileInfo.fileName.keyword"
          }
        },
        "agents_count": {
          "cardinality": {
            "field": "agentId.keyword"
          }
        },
        "first_file": {
          "top_hits": {
            "from": 0,
            "size": 1,
            "_source": {
              "includes": [
                "fileInfo.fileQualifier",
                "fileInfo.fileName",
                "fileInfo.fileDescription",
                "fileInfo.hash",
                "fileInfo.publisher",
                "fileInfo.fileSize"
              ],
              "excludes": []
            },
            "sort": [
              {
                "@timestamp": {
                  "order": "asc"
                }
              }
            ]
          }
        }
      }
    },
    "filtered_count": {
      "cardinality": {
        "field": "fileInfo.fileQualifier"
      }
    }
  }
}

Any trial I did for ordering by fileName was not successful, since the search engine doesn't support order by pipeline aggregations.
It ended with error like this:

  "error" : {
    "root_cause" : [
      {
        "type" : "aggregation_execution_exception",
        "reason" : "Invalid aggregation order path [file_name]. Buckets can only be sorted on a sub-aggregator path that is built out of zero or more single-bucket aggregations within the path and a final single-bucket or a metrics aggregation at the path end."
      }
    ],

Describe the solution you'd like
I'd like to be able to do:

        "firstFilename": {
          "min": {
            "field": "fileInfo.fileName.keyword"
          }

and then order the result by 'firstFilename'.
Alternatively, use another function such as FIRST/LAST instead of min/max.

Describe alternatives you've considered
Transform index might have been a solution. However, it is a lot more complicated and also doesn't support all types of aggregations like top_hits and other complex queries.

Additional context
Here is how FIRST/LAST has been implemented in SQL engine, therefore it is surely possible:
https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-functions-aggs.html#sql-functions-aggs-first
The issue is also referenced here:
https://github.com/elastic/elasticsearch/issues/37198

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source files, tests, or implementation entry points are named. Start from the aggregation query and the reported invalid aggregation order path error, then review how metric aggregations and terms-bucket ordering are represented; done means a keyword-field FIRST/LAST-style aggregation can be used to order aggregation results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.