[BUG] Object-vs-scalar mapping conflict across a wildcard resolves nondeterministically (last-write-wins)

Query Information

PPL Command/Query:

source=mc_* | fields `labels.zone` | head 1

Expected Result: the same resolved type for labels.zone on every node and every request.

Actual Result: the type is string on some nodes and struct on others, for the same indices and the same data. Running the query against six freshly started single-node clusters with an identical fixture:

  run 1 -> labels.zone : struct
  run 2 -> labels.zone : string
  run 3 -> labels.zone : string
  run 4 -> labels.zone : struct
  run 5 -> labels.zone : string
  run 6 -> labels.zone : string

Anything that depends on the resolved type therefore behaves inconsistently. Most visibly, timechart ... by \labels.zone`returns a chart when the merge lands onstringand fails withCannot chart by [labels.zone] because it is an object.(#5750, #5751) when it lands onstruct`.

Dataset Information

Dataset/Schema Type

• OpenTelemetry (OTEL)
• Simple Schema for Observability (SS4O)
• Open Cybersecurity Schema Framework (OCSF)
• Custom (details below)

Two indices matched by one wildcard, disagreeing on whether labels.zone is a scalar or an object. This happens in practice when a mapping is changed at a rollover boundary: older backing indices carry the object form, newer ones a flat keyword.

Index Mapping — mc_scalar:

{"mappings": {"properties": {
  "@timestamp": {"type": "date"},
  "labels": {"properties": {"zone": {"type": "keyword"}}}
}}}

mc_object:

{"mappings": {"properties": {
  "@timestamp": {"type": "date"},
  "labels": {"properties": {"zone": {"properties": {"name": {"type": "keyword"}}}}}
}}}

Sample Data

// mc_scalar
{"@timestamp": "2026-01-01T00:01:00Z", "labels": {"zone": "z1"}}
// mc_object
{"@timestamp": "2026-01-01T00:02:00Z", "labels": {"zone": {"name": "z2"}}}

Bug Description

Issue Summary

When a wildcard spans indices that map a path as an object in some and a scalar in others, the merged type is decided by whichever index happens to be iterated last. That order is randomized per JVM, so the resolved type varies between nodes and changes when a node restarts.

Steps to Reproduce

# 1. Two indices disagreeing on labels.zone (mappings above)
curl -s -XPUT 'localhost:9200/mc_scalar' -H 'Content-Type: application/json' -d '{
 "mappings": {"properties": {"@timestamp": {"type": "date"},
   "labels": {"properties": {"zone": {"type": "keyword"}}}}}}'
curl -s -XPUT 'localhost:9200/mc_object' -H 'Content-Type: application/json' -d '{
 "mappings": {"properties": {"@timestamp": {"type": "date"},
   "labels": {"properties": {"zone": {"properties": {"name": {"type": "keyword"}}}}}}}}'

curl -s -XPOST 'localhost:9200/mc_scalar/_doc?refresh=true' -H 'Content-Type: application/json' \
  -d '{"@timestamp": "2026-01-01T00:01:00Z", "labels": {"zone": "z1"}}'
curl -s -XPOST 'localhost:9200/mc_object/_doc?refresh=true' -H 'Content-Type: application/json' \
  -d '{"@timestamp": "2026-01-01T00:02:00Z", "labels": {"zone": {"name": "z2"}}}'

curl -s -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' \
  -d '{"transient":{"plugins.calcite.enabled":true}}'

# 2. Observe the resolved type
curl -s -XPOST 'localhost:9200/_plugins/_ppl' -H 'Content-Type: application/json' \
  -d '{"query":"source=mc_* | fields `labels.zone` | head 1"}'

# 3. Restart the node (or repeat against a different node) and run step 2 again
#    -> the schema type alternates between "string" and "struct"

Impact

• The same query succeeds or fails depending on which coordinating node serves it, and can flip after a node restart or when the matched index set changes at a rollover. That is hard to report, hard to reproduce, and looks to a user like an intermittent bug in their dashboard.
• When the field is a scalar in the majority of indices — the common shape after a mapping change — there is no way to chart or aggregate it reliably, even though the data is there.

Root Cause

Merging happens per index in OpenSearchDescribeIndexRequest.getFieldTypes:110-127:

Map<String, IndexMapping> indexMappings = client.getIndexMappings(...);
for (IndexMapping indexMapping : indexMappings.values()) {
  MergeRuleHelper.merge(fieldTypes, deepCopy(indexMapping.getFieldMappings()));
}

MergeRuleHelper tries its rules in order (MergeRuleHelper:13):

Rule	Matches	Object vs scalar?
DeepMergeRule	both sides the same ExprCoreType (STRUCT or ARRAY)	no — the core types differ
TextKeywordConflictRule	text vs keyword, or text with/without a keyword sub-field	no — neither side is text
LatestRule	everything	yes — target.put(key, source), last write wins

So the outcome is decided by the iteration order of the map returned by client.getIndexMappings. OpenSearchNodeClient.getIndexMappings:101-104 builds it with Collectors.toUnmodifiableMap, and the JDK deliberately randomizes iteration order of its immutable maps using java.util.ImmutableCollections.SALT, which is seeded once per JVM start. Same keys, same code, different process → different order.

Reduced to the merge decision alone, with 11 index names of which 4 map the path as an object:

  out of 20 JVM runs:   "object" won 15x
                        "keyword" won  5x

The order is stable within a process (the mappings are re-fetched per query, but SALT is fixed for the life of the JVM), which is why this presents as "it works on one node and not another" rather than as flapping on consecutive requests.

Proposed Fix

1. A deterministic rule for the conflict. Add ObjectScalarConflictRule next to TextKeywordConflictRule, registered ahead of LatestRule:

private static final List<MergeRule> RULES =
    List.of(new DeepMergeRule(),
            new TextKeywordConflictRule(),
            new ObjectScalarConflictRule(),   // new
            new LatestRule());                // must come last

It matches when one side is Object/Nested and the other a scalar, and resolves to the scalar side. The scalar is the only side with a value that can be grouped, sorted or charted, and it keeps doc-values pushdown available — resolving to text instead (what TextKeywordConflictRule does for its case) would force _source retrieval and a full scan.

2. Let partial-result mode engage for this conflict. PartialResultAggregatePushdown.resolveBucketSignature:107-121 returns a t:<MappingType> token for an object-mapped index, so plan() sees two "aggregatable" signature groups and bails at if (aggregatableGroups.size() != 1) return null. Returning null for Object/Nested, as it already does for the text family, puts those indices in excludedIndices, and CalciteLogicalIndexScan.tryPartialResultAggregate then narrows the aggregation to the indices that have the leaf and attaches a warning naming the rest.

Nothing is needed on the decode side: the scalar branch of OpenSearchExprValueFactory.parse already catches an object value and returns null (#5618), and the mirror case is guarded by #5685.

Open question for (2): with the merged type resolved to the scalar, does a pushed-down composite terms source on that path error on the shards where it is an object, or silently contribute nothing? If it errors, the exclusion has to apply regardless of the opt-in plugins.query.partial_result.on_mapping_conflict.enabled setting. An object-typed path has no leaf field at all, so "absent in those indices" — which a wildcard already tolerates — is arguably the honest semantic here, unlike the text/keyword case where the data exists but is not aggregatable.

Environment Information

OpenSearch Version: main, with plugins.calcite.enabled: true. The merge code is not Calcite-specific, so the resolved type is nondeterministic on the legacy path too.

Additional Details

• #5750 / #5751 — the 500 that this conflict caused when the merge landed on the object side, now an actionable 400. That fix stops the crash but cannot make the field usable; this issue is what makes it usable.
• #5685, #5618 — the value-decode side of the same conflict.
• #5610 — the broader schema-conflict policy discussion this belongs to.
• #4659 — TextKeywordConflictRule, the existing precedent for resolving a mapping conflict deterministically rather than by last-write-wins.

Start with OpenSearchDescribeIndexRequest.getFieldTypes and MergeRuleHelper, then inspect OpenSearchNodeClient.getIndexMappings and PartialResultAggregatePushdown.resolveBucketSignature. Trace the existing conflict rules and partial-result path first. Done means object-versus-scalar merging is deterministic, scalar fields remain usable, and object-mapped indices receive the intended partial-result handling and warning.