opensearch-project / opensearch-project/sql

[BUG] Projecting a nested subfield returns only the first child

Open
#5,748 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug PPL
Dominant language
Java
Stars
176
Forks
229
Avg merge
2d 21h
Merged PRs (30d)
43

Description

Query Information

PPL Command/Query:

source=nested_repro | fields id, events.name, events.status

Expected Result:
6 rows — one per events child (doc1:1, doc2:2, doc3:1, doc4:2).

Actual Result:
4 rows — one per parent document, containing only the first child:

['doc1', 'db_query',  'ok']
['doc2', 'db_query',  'error']     <- child {http_call, ok} dropped
['doc3', 'http_call', 'error']
['doc4', 'http_call', 'error']     <- child {db_query, ok} dropped

The same engine is internally inconsistent — stats sees all children while fields does not:

source=nested_repro | stats count() by events.name

returns db_query: 3, http_call: 3 (correct, matches a nested terms agg), yet only 2 projected rows ever contain db_query.

Dataset Information

Dataset/Schema Type

  • OpenTelemetry (OTEL)
  • Simple Schema for Observability (SS4O)
  • Open Cybersecurity Schema Framework (OCSF)
  • Custom (details below)

Index Mapping

{
  "mappings": {
    "properties": {
      "id": {"type": "keyword"},
      "events": {
        "type": "nested",
        "properties": {
          "name": {"type": "keyword"},
          "status": {"type": "keyword"}
        }
      }
    }
  }
}

Sample Data

{"id":"doc1","events":[{"name":"db_query","status":"ok"}]}
{"id":"doc2","events":[{"name":"db_query","status":"error"},{"name":"http_call","status":"ok"}]}
{"id":"doc3","events":[{"name":"http_call","status":"error"}]}
{"id":"doc4","events":[{"name":"http_call","status":"error"},{"name":"db_query","status":"ok"}]}

Bug Description

Issue Summary:
When a nested subfield is projected, only element [0] of the array is read. Remaining children are dropped with no warning, error, or row multiplication. Because the value is typed as a flat scalar rather than a collection, there is no way for a user to reach the other children.

Note: this is not the documented behavior of plugins.query.field_type_tolerance. That setting defaults to true ("preserve arrays") and does work on field roots — flipping it changes the output of fields events — but dotted subfield paths like events.name collapse to the first element in both states, so the setting is never consulted here.

Steps to Reproduce:

  1. Create the index and load the 4 documents above.
  2. PUT _cluster/settings {"transient":{"plugins.calcite.enabled":true}}
  3. Run the query — 4 rows, first child only.
  4. Compare to _source, which holds 6 children total.

Root cause:
core/src/main/java/org/opensearch/sql/data/model/ExprValueUtils.java:230, unchanged since #3476:

public static ExprValue resolveRefPaths(ExprValue value, List<String> paths) {
  ExprValue wholePathValue = value.keyValue(String.join(PATH_SEP, paths));
  // For array types only first index currently supported.
  if (value.type().equals(ExprCoreType.ARRAY)) {
    wholePathValue = value.collectionValue().getFirst().keyValue(paths.getFirst());
  }

This is reached from OpenSearchIndexEnumerator.resolveForCalcite. #3476 deliberately chose to flatten nested fields into scalar columns ("option1") over a PartiQL-like scoped approach (#3459), and the first-child shortcut is a direct consequence.

Impact:
Silently incomplete results whenever a nested subfield is projected — data loss with no signal to the user, and results that contradict stats on the same field in the same engine. Legacy SQL returns all children via nested(message.info); PPL Calcite has no equivalent.

Environment Information

OpenSearch Version: 3.9.0-SNAPSHOT (./gradlew run on main @ 96399c590)

Additional Details:

  • Calcite (v3) engine, plugins.calcite.enabled=true
  • Related: #5333 (array-atomicity umbrella) covers the same shortcut for plain arrays; this issue is the nested-type manifestation
  • Also causes the false negative in the companion nested-where correlation bug when pushdown is disabled
  • Tracked under #4625

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with core/src/main/java/org/opensearch/sql/data/model/ExprValueUtils.java:230 and trace how OpenSearchIndexEnumerator.resolveForCalcite uses resolveRefPaths. Reproduce the PPL query against the four-document dataset, then verify that projecting events.name and events.status returns all six child rows without dropping later nested elements.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.