opensearch-project / opensearch-project/sql
[BUG] PPL FIRST/LAST/TAKE appear to ignore a preceding sort on multi-shard indices
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 176
- Forks
- 229
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 43
Description
What is the bug?
On a multi-shard index, FIRST(), LAST(), and TAKE() appear not to preserve a preceding PPL sort.
The same query returns the expected values on a one-shard index. earliest() and latest() remain correct over the same documents.
Reproduction
Create identical one-shard and five-shard indices:
for SH in 1 5; do
curl -XDELETE "localhost:9200/rp_$SH"
curl -XPUT "localhost:9200/rp_$SH" -H 'Content-Type: application/json' -d "{
\"settings\":{\"number_of_shards\":$SH,\"number_of_replicas\":0},
\"mappings\":{\"properties\":{\"@timestamp\":{\"type\":\"date\"},\"txt\":{\"type\":\"keyword\"}}}}
}"
curl -XPOST "localhost:9200/rp_$SH/_bulk?refresh=true" \
-H 'Content-Type: application/x-ndjson' --data-binary '
{"index":{"_id":"1"}}
{"@timestamp":"2024-01-01T10:00:00Z","txt":"a"}
{"index":{"_id":"2"}}
{"@timestamp":"2024-01-02T10:00:00Z","txt":"b"}
{"index":{"_id":"3"}}
{"@timestamp":"2024-01-03T10:00:00Z","txt":"c"}
'
done
Run against each index:
source=rp_N
| sort @timestamp
| stats FIRST(@timestamp), LAST(@timestamp), TAKE(@timestamp, 2),
earliest(@timestamp), latest(@timestamp)
Observed results:
1 shard: FIRST=01-01 LAST=01-03 TAKE=[01-01,01-02] earliest=01-01 latest=01-03
5 shards: FIRST=01-03 LAST=01-03 TAKE=[01-03,01-02] earliest=01-01 latest=01-03
The five-shard result is repeatable, but does not follow the explicit timestamp ordering.
Existing integration-test failures
The existing alias-field tests show the same issue with different shard placement:
CalciteAliasFieldAggregationIT::testFirstWithAliasField
query: source=%s | sort @timestamp | stats FIRST(@timestamp)
expected: 2024-01-01 10:00:00
actual: 2024-01-02 10:00:00
CalciteAliasFieldAggregationIT::testLastWithAliasField
query: source=%s | sort @timestamp | stats LAST(@timestamp)
expected: 2024-01-03 10:00:00
actual: 2024-01-02 10:00:00
CalciteAliasFieldAggregationIT::testTakeWithAliasField
query: source=%s | sort @timestamp | stats TAKE(@timestamp, 2)
expected: [2024-01-01, 2024-01-02]
actual: [2024-01-01, 2024-01-03]
Expected behavior
After sort @timestamp, FIRST() and LAST() should return the first and last values in that order, independent of shard count.
For TAKE(), the documentation currently says result order is not guaranteed. However, it would be useful to clarify whether the preceding sort should at least determine which values are selected.
Possible implementation clue
This may be related to how these aggregations are pushed down as top_hits:
FIRST -> createTopHitsBuilder(..., 1, true, false, null, null)
TAKE -> createTopHitsBuilder(..., size, true, false, null, null)
LAST -> createTopHitsBuilder(..., 1, true, true, "_doc", SortOrder.DESC)
FIRST and TAKE do not pass a sort key, while LAST uses _doc, which is not a globally comparable document-order key across shards. The preceding PPL collation may therefore not reach the generated top_hits aggregation. This is only a possible explanation and needs maintainer confirmation.
This also appears consistent with the limitation noted in #5537: adding an explicit sort does not stabilize stats first()/last()/take() on multi-shard execution.
Environment
- OpenSearch 3.7.0
opensearch-sqlplugin- Single node
- One versus five primary shards
- No replicas
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the existing CalciteAliasFieldAggregationIT tests named testFirstWithAliasField, testLastWithAliasField, and testTakeWithAliasField, then trace the FIRST, LAST, and TAKE top_hits generation described in the issue. Compare one-shard and multi-shard execution and verify that the preceding PPL sort is respected; the tests should pass with the expected values, with TAKE semantics clarified if its ordering remains unspecified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100