Default sort has no tie-break: documents sharing a moddate reorder when reads switch from Elasticsearch to OpenSearch

Problem Statement

When a query arrives with no sortBy, both index-operations classes inject the same default sort before sending it to the engine:

// ContentFactoryIndexOperationsES:296
searchSourceBuilder.sort("moddate", SortOrder.DESC);

// ContentFactoryIndexOperationsOS:251
searchRequestBuilder.sort(SortOptions.of(so -> so.field(FieldSort.of(fs -> fs
        .field("moddate").order(SortOrder.Desc).unmappedType(FieldType.Date)))));

That default is what keeps $dotcontent.pull with no sortBy stable across the ES→OpenSearch migration, and it works. But it sorts on a single non-unique field and supplies no tie-break. Documents sharing a moddate to the millisecond fall back to internal Lucene document order, which is a function of how each index was physically written. Elasticsearch and OpenSearch are populated by independent write paths, so their segment layouts differ and the tie resolves differently in each engine.

The sort is hardcoded at both sites — there is no configuration key that can add a secondary criterion, so an operator hitting this has no workaround short of editing every affected template.

Impact. Reads switch engines at Phase 2. Any listing whose items share a moddate — bulk-imported content, content created by the same automated process, anything published in a batch — can come back in a different sequence after the switch. Same items, same count, different order.

Why it is easy to miss. Nothing count-based detects it. Both indices hold the same documents, so /api/v1/index/migration/readiness reports IN_SYNC, driftPercent: 0.0 and safeToAdvance: true. The first person to notice is an editor looking at a page, which is the most expensive place to find it.

Measurements

Measured on a real-data instance in Phase 3, both engines fully in sync (identical doc counts, drift 0.0%), comparing the hit-id sequence returned by the same query against each engine's index directly. Content types anonymised.

Two things to read from this:

1. The injected default does its job. Every case matches once it is applied — which is why this is a latent gap, not a live regression for the content path. Filing it before it bites rather than after.
2. The one query that carried moddate ties still matched, i.e. today the tie happens to resolve the same way in both engines. Nothing guarantees that survives a segment merge, a reindex, or a restore.

Raw-query paths (/api/es/raw, /api/es/search) receive no injected default at all and are exposed to the first column, but there the caller owns the sort and that is the documented contract.

Steps to Reproduce

Requires an instance in Phase 1 or later with Elasticsearch and OpenSearch in sync.

1. Index several contentlets of the same content type that share an identical moddate (a bulk import, or a script that publishes them in one batch, produces this naturally).
2. Confirm the two engines agree on content: GET /api/v1/index/migration/readiness reports IN_SYNC with driftPercent: 0.0.
3. Send the same query to each engine's index directly, with the sort dotCMS injects, and compare the returned _id sequence:

BODY='{"query":{"term":{"contenttype":"<type>"}},"size":20,
       "sort":[{"moddate":{"order":"desc","unmapped_type":"date"}}],
       "_source":["identifier","moddate"]}'

curl -s -XPOST "$ES/<cluster>.live_<ts>/_search"     -H 'Content-Type: application/json' -d "$BODY"
curl -s -XPOST "$OS/<cluster>.live_<ts>.os/_search"  -H 'Content-Type: application/json' -d "$BODY"

4. Among the documents sharing a moddate, the relative order is decided by document order and is not guaranteed to agree between the two engines.

Expected: the same query against two indices holding the same documents returns the same sequence, regardless of engine.

Actual: documents tied on moddate are ordered by segment layout, which differs per engine.

Acceptance Criteria

• The default sort applied when sortBy is empty includes a unique secondary criterion (e.g. identifier), so the returned order is fully determined by the query rather than by segment layout.
• The change is applied at both sites — ContentFactoryIndexOperationsES and ContentFactoryIndexOperationsOS — so the two engines cannot drift apart.
• An unsorted query returns the same hit sequence from Elasticsearch and OpenSearch when both indices hold the same documents, including documents sharing a moddate.
• The order is stable across a segment merge and a full reindex, not merely equal on a freshly built index.
• Existing behaviour is preserved for queries that do supply a sortBy: the caller's criteria still take precedence and the tie-break is appended, never prepended.
• Regression integration test that indexes several documents with an identical moddate and asserts a stable, engine-independent order.

dotCMS Version

Reproduced against a build of main (dotcms-core-1.0.0-SNAPSHOT) running Elasticsearch 7.10.2 and OpenSearch 3.4.0. The affected code is present in both the ES and OS index-operations classes, so it applies to every version carrying the OpenSearch migration work.

Severity

Medium - Some functionality impacted

Links

NA