opensearch-project / opensearch-project/sql
[BUG] PPL returns HTTP 200 with silently partial results when shards fail (no _shards failure surfaced)
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 176
- Forks
- 229
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 43
Description
Describe the bug
When shards fail (e.g. Workload Management rejects queries at the shard level, or circuit-breaker/timeout on some shards), OpenSearch returns HTTP 200 with a _shards block such as {total:1000, successful:12, failed:988}. PPL discards the _shards info and returns the partial result as if it were complete — no error, no warning. Users get silently incorrect answers during load spikes (e.g. a returned count of 50 might really be 5000). DSL surfaces the shard failures; PPL does not.
To Reproduce
Run a PPL aggregation on a large multi-shard index under enough load (or WLM CPU enforcement) that some shards are rejected. Compare the PPL count to the equivalent _search DSL response's _shards.failed. PPL reports only the partial count with no indication.
Expected behavior
When failedShards > 0, PPL should either (a) attach a warning naming the shard-failure counts (total/successful/failed), or (b) optionally fail, so the consumer knows the answer is partial.
Root cause
OpenSearchResponse (opensearch module) ingests the SearchResponse but reads only getHits() and getAggregations() — it discards getTotalShards(), getSuccessfulShards(), getFailedShards(), getShardFailures(). The request also does not set allowPartialSearchResults(false), so it inherits the cluster default (true). Identical on the V2 and Calcite scan paths — both funnel through OpenSearchResponse.
Proposed fix
Capture failedShards/successfulShards/totalShards off the SearchResponse in OpenSearchResponse and, when failedShards > 0, raise a structured warning through the response warning channel (the Warning{type,message,detail} → QueryResponse.warnings → SimpleJsonResponseFormatter path added for mapping-conflict partial results). Optionally add a per-request/cluster setting to hard-fail instead of warn (mirrors DSL allow_partial_search_results).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Trace how OpenSearchResponse ingests SearchResponse for both the V2 and Calcite scan paths, then follow the Warning to QueryResponse.warnings and SimpleJsonResponseFormatter path used for mapping-conflict partial results. Capture the shard counts and surface failed-shard information when failures occur; done means PPL no longer presents partial results without an indication.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100