A reindex at phase 3 splits the engines onto different index generations: deprecated search returns null, readiness report goes blind and reports health

Found during a lab run of the ES→OpenSearch 3 migration against
docker/docker-compose-examples/single-node-os-migration, by Jamie Mauro.

Description

A full reindex run at phase 3 leaves the two engines on different index generations. dotCMS
repoints to the new OpenSearch name; Elasticsearch keeps the old one. Two things break as a result,
and both are silent.

Observed after the reindex:

Engine	Indexes
source (opensearch1)	working_20260916231043 (8 docs) · live_20260916231043 (6) — old generation only
target (opensearch3)	working_20260918174350.os (8) · live_20260918174350.os (6) — new working_20260916231043.os (6) · live_20260916231043.os (4) — orphaned

Both engines were written — the counts match — but under different index names. The name divergence
is the defect, not the data.

Symptom 1 — the deprecated search path returns null, and raw Velocity reaches the page

A probe template at phase 3, everything reporting IN_SYNC:

search()   -> 5
esSearch() -> $old.size()

$old.size() rendered as literal text. In Velocity that means
#set($old = $estool.esSearch($q)) assigned nothing — the call returned null rather than throwing,
so the template carried on and emitted unresolved source into the response body.

esSearch() resolves the active index name through the pointer records, asks Elasticsearch for
…20260918174350, and finds nothing there. Empty result, not an error.

That is a third distinct failure mode for this path, and the worst of the three:

Trigger	Behaviour	What a visitor sees
Malformed query	throws	blank page
Source engine unreachable	throws	blank page
After a phase-3 reindex	returns null	$old.size() printed into the page

The log window covering the probe shows no exception and no ERROR line — an empty result from a
missing index is indistinguishable from a legitimate zero-hit search.

Symptom 2 — the readiness report goes blind, and reports health because of it

Same state, immediately after:

$ curl … /api/v1/index/migration/readiness | jq '.content'
{}

No LIVE, no WORKING, no counts, no physicalName. Yet the verdict in the same response:

"outOfSyncCount": 0,
"safeToAdvance": true,
"summary": "Phase 3 (OpenSearch only) — the final phase, nothing to advance to.
            No index shows Elasticsearch behind OpenSearch; still verify before any downgrade."

The all-clear is vacuous. outOfSyncCount: 0 because there are zero entries to count; "no index
shows Elasticsearch behind OpenSearch" is trivially true when the report holds no indexes.

The comparison that isolates the cause: at phase 3, before the reindex, content was populated and
reported outOfSyncCount: 2. Same phase, same engines. The reindex is what emptied it.

Why this matters

An operator repairs drift, re-runs the report, watches outOfSyncCount drop 2 → 0, and reasonably
concludes the repair worked. It did — but the report's agreement was coincidence. It would have said
exactly the same thing had the repair failed, because it had stopped looking.

It also disarms the documented safety gate. The operator runbook's entry condition for the final
stage is "the readiness report still says IN_SYNC with outOfSyncCount: 0". In this state that
check passes on empty data — the gate designed to stop an unsafe advance waves you through.

And a full reindex is the runbook's own recommended repair at phase 3, so the recovery action puts
the installation into the state where problems can no longer be detected.

Likely mechanism

The reconciler pairs indexes across engines by name. With the two engines on different generations
(20260916231043 vs 20260918174350.os) it appears unable to form pairs at all, so it emits nothing
rather than reporting a mismatch — and an empty collection then reads as "nothing out of sync".

Worth checking as part of this: nothing cleaned up the superseded .os indexes, which remain on the
target engine orphaned.

Acceptance Criteria

• A reindex at phase 3 leaves both engines on the same index generation, or the divergence is
  recorded somewhere the reconciler can follow.
• esSearch() / esRaw() pointed at an index that does not exist on their engine raise rather
  than return null — a null return is what puts raw Velocity on the page.
• The readiness report never returns an empty content map with a green verdict. If it cannot
  form index pairs, it says so and safeToAdvance is false.
• outOfSyncCount: 0 is distinguishable from "nothing was measured", in both the field and the
  summary text.
• Superseded .os indexes are cleaned up, or their presence is reported.
• Regression test: phase 3, full reindex, assert content is populated afterwards.

Additional Context

Same lab run also produced #37636 (the source engine being switched off), #37637 (PhaseRouter
misclassifying a malformed query) and #37638 (readiness summary wording). This one and #37636
compound: both make the readiness report unusable at the terminal phase, by different routes.