opensearch-project / opensearch-project/sql

[FEATURE] Improve COUNT function for multivalued fields

Open
#3,205 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug SQL
Dominant language
Java
Stars
176
Forks
229
Avg merge
2d 21h
Merged PRs (30d)
43

Description

Is your feature request related to a problem?
When the COUNT function is applied to a multivalued field, it will return the count of all scalar values associated with that field. Consider this table:

x | y
--+----------
1 | {1, 2, 3}
2 | {4, 5}
3 | NULL

COUNT(y) will return 5. Most SQL engines would return 2 in this case, since there are 2 rows containing a value for y.

What solution would you like?
The aggregate requested from the OpenSearch server could be more like:

  "aggs": {
    "foo": {
      "filter": {
        "bool": {
          "must": [
            {
              "exists": {
                "field": "y"
              }
            }
          ]
        }
      }
    }
  }

There are a few forms of COUNT that need to be considered:

  • COUNT(<FIELD>) - can use an aggregate like above
  • COUNT(<LITERAL>) - can use an aggregate like above, but use the field _id to count all rows
  • COUNT(*) - can use an aggregate like above, but use the filed _id to count all rows
  • COUNT(NULL) - will always return 0

What alternatives have you considered?
The alternative is to keep the existing behaviour. This is likely to produce results that are unexpected for users, especially if they are used to other SQL engines.

Do you have any additional context?
Recently a change was made to support multivalued fields in the SQL plugin. Some of the aggregates may need to be updated to handle the new multivalued fields.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No file or test is named. Start by locating the SQL plugin's COUNT aggregate handling and the recent multivalued-field support, then trace how field, literal, wildcard, and NULL expressions are represented. Done means COUNT(field) counts rows with a value, COUNT(literal) and COUNT(*) count rows, and COUNT(NULL) returns zero, with coverage for each form.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
backend, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.