opensearch-project / opensearch-project/sql
[BUG] PPL subsearch maxout settings do not apply on the Analytics Engine path (diverges at defaults)
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 176
- Forks
- 229
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 43
Description
What is the bug?
On the Analytics Engine (unified query) path, PPL subsearches run unbounded, while the default (non-AE) pipeline caps them. Unlike the other settings-fidelity gaps in this area, this one diverges at defaults — no operator configuration is required to hit it.
| Setting | Cluster default | AE path value |
|---|---|---|
plugins.ppl.subsearch.maxout |
10000 |
0 (unlimited) |
plugins.ppl.join.subsearch_maxout |
50000 |
0 (unlimited) |
Verified on a default AE context:
SysLimit[querySizeLimit=10000, subsearchLimit=0, joinSubsearchLimit=0]
SysLimit.fromSettings reads both keys, and 0 means "no LogicalSystemLimit injected", so nothing bounds the subsearch.
Root cause — and why this one is a design question, not a plain bug
This is deliberate, and documented in UnifiedQueryContext.Builder:
Settings.Key#PPL_JOIN_SUBSEARCH_MAXOUTdefaults to0to avoid injectingLogicalSystemLimitinto the logical plan, which is an OpenSearch-specific operational concern irrelevant to external consumers of the unified query API.Settings.Key#PPL_SUBSEARCH_MAXOUTis set to0for the same reason.
That rationale holds for external consumers of the unified query API as a library (e.g. planning PPL for a non-OpenSearch execution target), where an OpenSearch operational cap is indeed meaningless.
It does not obviously hold for the in-cluster REST path, where the query is being served by an OpenSearch node, consumes that node's memory, and the operator has configured a cap expecting it to apply. Two queries against the same cluster get different memory-safety guarantees purely based on whether the target index happens to use the composite data format.
What is the expected behavior?
Needs a maintainer decision between:
Option A — forward from the REST handler. Add both keys to RestUnifiedQueryAction.FORWARDED_CLUSTER_SETTINGS. This preserves the library default of "unlimited" for external API consumers (who never go through the REST handler) while restoring cluster fidelity in-cluster. Minimal change, and it keeps the documented rationale intact for the case it was written for.
Caveat: this is a behavior change at defaults — AE subsearches would go from unbounded to capped at 10000 / 50000. Queries relying on the current unbounded behavior would start hitting the cap. Needs a release note.
Option B — keep as-is. Accept that AE subsearches are unbounded by design, and document the divergence explicitly so operators know the cap does not apply to composite indices.
Do you have any additional context?
Found while fixing the same class of defect in #5611 (settings the AE path silently ignored). That PR forwards plugins.query.size_limit, plugins.ppl.pattern.*, and plugins.ppl.values.max.limit, and explicitly excludes these two keys pending this decision — the exclusion is pinned by a test (everySeededPlanningSettingIsClassified) so it stays a conscious choice rather than drift.
Related: #5734 (plugins.calcite.all_join_types.allowed guardrail inactive on the AE path).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the documented defaults in UnifiedQueryContext.Builder and the classification test everySeededPlanningSettingIsClassified, then inspect RestUnifiedQueryAction.FORWARDED_CLUSTER_SETTINGS and the context from #5611. The issue is done when maintainers choose whether REST requests forward both settings or the divergence is explicitly documented, with the relevant test and release-note decision captured.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100