opensearch-project / opensearch-project/sql

[BUG][Umbrella] Array values are not treated as atomic values in SQL/PPL operations

Open
#5,333 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug PPL
Dominant language
Java
Stars
176
Forks
229
Avg merge
2d 21h
Merged PRs (30d)
43

Description

What is the bug?

After #1300 enabled array values in query evaluation and result sets, the engine decomposes arrays into individual scalar elements for operations that should treat the array as a single unit. This causes incorrect results or exceptions across multiple SQL/PPL operations.

Affected operations

Operation Symptom Issue
WHERE comparisons (=, !=, >, >=, <, <=) If any element passes, the whole array passes the filter #3136
ORDER BY Only the first element is used for comparison #3133
GROUP BY Each array element is treated as a separate group key #3132
MAX/MIN aggregation Operates on individual elements across all arrays instead of comparing whole arrays #3138
regex command Only matches patterns against the first element of an array field #4461
Calcite eval/arithmetic expressions ClassCastException: ArrayList cannot be cast to Long — no array handling at all #5232

How to reproduce

Create an index with array-valued fields:

{"x": 1, "y": [1, 2]}
{"x": 2, "y": [3, 4]}
{"x": 3, "y": [1, 5]}
{"x": 4, "y": [1, 2]}
{"x": 5, "y": [2, 3]}

Then run any of the following:

-- WHERE: returns rows where ANY element > 3, not treating array as atomic
SELECT x, y FROM test WHERE y > 3;

-- ORDER BY: sorts by first element only
SELECT x, y FROM test ORDER BY y;

-- GROUP BY: explodes array into individual group keys
SELECT COUNT(x), y FROM test GROUP BY y;

-- MAX/MIN: returns max/min of all individual elements, not max/min array
SELECT MAX(y) FROM test;

Expected behavior

Arrays should be treated as indivisible values. Operations on array vs scalar should either:

  • Throw an exception (type mismatch), or
  • Have well-defined semantics (e.g., element-wise comparison for array vs array)

Root cause

The engine does not distinguish between array values and scalar values at the operator level. When an array is encountered, it is implicitly flattened or only the first element is used, depending on the code path.

Duplicates

This issue consolidates the following duplicates:

  • #3132 — GROUP BY with array values
  • #3133 — ORDER BY with array values
  • #3136 — Comparison operators with array values
  • #3138 — MAX/MIN with array values
  • #4461 — regex matching on array fields
  • #5232 — ClassCastException on multi-valued long fields
  • #3206 — Parse on multi-valued fields
  • #4365 — Pattern on multi-valued fields

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the affected-operation issues listed in the table, especially #3132, #3133, #3136, #3138, #4461, and #5232, then run the SQL examples against array-valued fields. Done means the project defines and consistently applies atomic-array behavior for the listed WHERE, ORDER BY, GROUP BY, aggregation, regex, and expression paths without flattening or array-related exceptions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.