opensearch-project / opensearch-project/sql

[BUG] PPL subsearch maxout settings do not apply on the Analytics Engine path (diverges at defaults)

Open
#5,735 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

analytic-engine bug design PPL
Dominant language
Java
Stars
176
Forks
229
Avg merge
2d 21h
Merged PRs (30d)
43

Description

What is the bug?

On the Analytics Engine (unified query) path, PPL subsearches run unbounded, while the default (non-AE) pipeline caps them. Unlike the other settings-fidelity gaps in this area, this one diverges at defaults — no operator configuration is required to hit it.

Setting Cluster default AE path value
plugins.ppl.subsearch.maxout 10000 0 (unlimited)
plugins.ppl.join.subsearch_maxout 50000 0 (unlimited)

Verified on a default AE context:

SysLimit[querySizeLimit=10000, subsearchLimit=0, joinSubsearchLimit=0]

SysLimit.fromSettings reads both keys, and 0 means "no LogicalSystemLimit injected", so nothing bounds the subsearch.

Root cause — and why this one is a design question, not a plain bug

This is deliberate, and documented in UnifiedQueryContext.Builder:

Settings.Key#PPL_JOIN_SUBSEARCH_MAXOUT defaults to 0 to avoid injecting LogicalSystemLimit into the logical plan, which is an OpenSearch-specific operational concern irrelevant to external consumers of the unified query API. Settings.Key#PPL_SUBSEARCH_MAXOUT is set to 0 for the same reason.

That rationale holds for external consumers of the unified query API as a library (e.g. planning PPL for a non-OpenSearch execution target), where an OpenSearch operational cap is indeed meaningless.

It does not obviously hold for the in-cluster REST path, where the query is being served by an OpenSearch node, consumes that node's memory, and the operator has configured a cap expecting it to apply. Two queries against the same cluster get different memory-safety guarantees purely based on whether the target index happens to use the composite data format.

What is the expected behavior?

Needs a maintainer decision between:

Option A — forward from the REST handler. Add both keys to RestUnifiedQueryAction.FORWARDED_CLUSTER_SETTINGS. This preserves the library default of "unlimited" for external API consumers (who never go through the REST handler) while restoring cluster fidelity in-cluster. Minimal change, and it keeps the documented rationale intact for the case it was written for.

Caveat: this is a behavior change at defaults — AE subsearches would go from unbounded to capped at 10000 / 50000. Queries relying on the current unbounded behavior would start hitting the cap. Needs a release note.

Option B — keep as-is. Accept that AE subsearches are unbounded by design, and document the divergence explicitly so operators know the cap does not apply to composite indices.

Do you have any additional context?

Found while fixing the same class of defect in #5611 (settings the AE path silently ignored). That PR forwards plugins.query.size_limit, plugins.ppl.pattern.*, and plugins.ppl.values.max.limit, and explicitly excludes these two keys pending this decision — the exclusion is pinned by a test (everySeededPlanningSettingIsClassified) so it stays a conscious choice rather than drift.

Related: #5734 (plugins.calcite.all_join_types.allowed guardrail inactive on the AE path).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the documented defaults in UnifiedQueryContext.Builder and the classification test everySeededPlanningSettingIsClassified, then inspect RestUnifiedQueryAction.FORWARDED_CLUSTER_SETTINGS and the context from #5611. The issue is done when maintainers choose whether REST requests forward both settings or the divergence is explicitly documented, with the relevant test and release-note decision captured.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.