opensearch-project / opensearch-project/sql

[BUG] PPL returns HTTP 200 with silently partial results when shards fail (no _shards failure surfaced)

Open
#5,687 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug PPL
Dominant language
Java
Stars
176
Forks
229
Avg merge
2d 21h
Merged PRs (30d)
43

Description

Describe the bug

When shards fail (e.g. Workload Management rejects queries at the shard level, or circuit-breaker/timeout on some shards), OpenSearch returns HTTP 200 with a _shards block such as {total:1000, successful:12, failed:988}. PPL discards the _shards info and returns the partial result as if it were complete — no error, no warning. Users get silently incorrect answers during load spikes (e.g. a returned count of 50 might really be 5000). DSL surfaces the shard failures; PPL does not.

To Reproduce

Run a PPL aggregation on a large multi-shard index under enough load (or WLM CPU enforcement) that some shards are rejected. Compare the PPL count to the equivalent _search DSL response's _shards.failed. PPL reports only the partial count with no indication.

Expected behavior

When failedShards > 0, PPL should either (a) attach a warning naming the shard-failure counts (total/successful/failed), or (b) optionally fail, so the consumer knows the answer is partial.

Root cause

OpenSearchResponse (opensearch module) ingests the SearchResponse but reads only getHits() and getAggregations() — it discards getTotalShards(), getSuccessfulShards(), getFailedShards(), getShardFailures(). The request also does not set allowPartialSearchResults(false), so it inherits the cluster default (true). Identical on the V2 and Calcite scan paths — both funnel through OpenSearchResponse.

Proposed fix

Capture failedShards/successfulShards/totalShards off the SearchResponse in OpenSearchResponse and, when failedShards > 0, raise a structured warning through the response warning channel (the Warning{type,message,detail}QueryResponse.warningsSimpleJsonResponseFormatter path added for mapping-conflict partial results). Optionally add a per-request/cluster setting to hard-fail instead of warn (mirrors DSL allow_partial_search_results).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Trace how OpenSearchResponse ingests SearchResponse for both the V2 and Calcite scan paths, then follow the Warning to QueryResponse.warnings and SimpleJsonResponseFormatter path used for mapping-conflict partial results. Capture the shard counts and surface failed-shard information when failures occur; done means PPL no longer presents partial results without an indication.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.