opensearch-project / opensearch-project/sql

[BUG] Incorrect value of quotted nested fields aggregations in JSON export

Open
#2,529 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug SQL
Dominant language
Java
Stars
176
Forks
229
Avg merge
2d 21h
Merged PRs (30d)
43

Description

What is the bug?

When quoting the field passed to an aggregation and when this field is a nested field, a SQL query with the JSON format return an incorrect value (0 or null) for the aggregation instead of the actual result.

How can one reproduce the bug?

Given an index with documents with a nested (field_nested.c2) and a non-nested field (field_float), using the following two queries to perform some basic aggregations:

  • SELECT sum(`field_float`) AS `SUM(field_float)`, sum(`field_nested.c2`) AS `SUM(field_nested.c2)` FROM data1 LIMIT 5000;
  • SELECT sum(field_float) AS `SUM(field_float)`, sum(field_nested.c2) AS `SUM(field_nested.c2)` FROM data1 LIMIT 5000;

We expect the same result. But when requesting JSON documents, the first query produce wrong values:

First query result:

{    
    "took": 1,    
    "timed_out": false,    
    "_shards": {    
        "total": 1,    
        "successful": 1,    
        "skipped": 0,    
        "failed": 0    
    },    
    "hits": {    
        "total": {    
            "value": 3,    
            "relation": "eq"    
        },    
        "max_score": null,    
        "hits": []    
    },    
    "aggregations": {    
        "SUM(field_nested.c2)": {    
            "value": 0    
        },    
        "SUM(field_float)": {    
            "value": 0.6000000163912773    
        }    
    }    
} 

Second query result:

{    
    "took": 1,    
    "timed_out": false,    
    "_shards": {    
        "total": 1,    
        "successful": 1,    
        "skipped": 0,    
        "failed": 0    
    },    
    "hits": {    
        "total": {    
            "value": 3,    
            "relation": "eq"    
        },    
        "max_score": null,    
        "hits": []    
    },    
    "aggregations": {    
        "SUM(field_nested.c2)": {    
            "value": 6    
        },    
        "SUM(field_float)": {    
            "value": 0.6000000163912773    
        }    
    }    
} 

Note how aggregations."SUM(field_nested.c2)".value is 0 in the first example and 6 it the second.

What is the expected behavior?

Quoted and non-quoted nested fields in aggregations should return the same (correct) value.

What is your host/environment?

  • OS: Debian
  • Version 11
  • OpenSearch 2.12.0

Do you have any additional context?

  • For the bug to happen, the field must be nested and quoted;
  • Only the JSON format is affected, the output of other formats has the expected value;
  • Some aggregations (e.g. sum as in the example above) always return 0, but other (e.g. avg) always return null instead of the actual value.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the two quoted and unquoted SQL aggregation queries against the nested-field example, then compare their JSON aggregation values. Trace the JSON-format aggregation path and verify that quoted and unquoted nested fields produce the same sum and average results. Done means the affected aggregations return the actual nested-field values rather than 0 or null.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
backend, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.