Retiring the source engine: deprecated search fails with an unsearchable log line, and the readiness report stops responding

Found during a lab run of the ES→OpenSearch 3 migration against
docker/docker-compose-examples/single-node-os-migration, by Jamie Mauro.

Description

Switching the source engine off is the event that actually breaks the deprecated search path — not any
phase transition, since esSearch() / esRaw() reach APILocator.getEsSearchAPI() in every phase and
never touch the phase router. It is also the intended end state: the runbook has the operator retire
the old cluster after a cooling-off period at phase 3.

Two separate defects surface at that moment, and together they leave an operator with no way to
diagnose what happened.

1. The deprecated path fails with one context-free, unsearchable log line

With the source engine stopped, the entire log output for a failing esSearch() request was:

14:27:19.609  ERROR util.DotRestHighLevelClientProvider$1 - [host=https://opensearch1:9200]

No message. No exception class. No stack trace. No URL, template, or call site. A hostname in
brackets.

Compare the same template failing through the router, which produced a descriptive ERROR PhaseRouter line, a full DotStateException with stack trace, and a WARN ESContentTool carrying
the request URL, language, IP and user. The rich logging lives in the router, and this path bypasses
it.

It is also literally unsearchable. A targeted grep for
esSearch|NoNodeAvailable|ConnectException|Connection refused across 200 lines of output returned
nothing, because the line contains none of those words — nor any word an operator would think to
search for. It was found only by tailing the window blind.

Practical effect: retire the old cluster and every deprecated call site breaks at once, with
essentially no diagnostic signal pointing at why.

2. The readiness report cannot answer at all

With opensearch1 stopped, the endpoint returns:

{"message":"opensearch1: Name or service not known"}

No phase, no verdict, no content — the whole report is gone. The stack trace shows why:

ESIndexAPI.getIndicesStats(ESIndexAPI.java:135)
  ContentIndexMirrorReconciler.statuses(ContentIndexMirrorReconciler.java:101)
  MigrationReadinessService.evaluate(MigrationReadinessService.java:56)
  MigrationReadinessResource.readiness(MigrationReadinessResource.java:109)

evaluate() calls the reconciler, which calls ESIndexAPI.getIndicesStats() unconditionally. One
unreachable engine takes down the entire endpoint rather than degrading to "OpenSearch state is X,
Elasticsearch unavailable".

Why this matters

The operator documentation calls this report the only reliable view of migration state, and has the
operator reading it at phase 3 — in the closing checks and throughout troubleshooting. Phase 3 is also
the state in which the source engine is meant to be decommissioned.

So the primary instrument stops working at exactly the point the migration is supposed to end, and the
one other signal available — the log — carries nothing usable. There is a window where the report
still works: phase 3 with the old cluster left running. The failure arrives whenever someone finally
switches that cluster off, by which time nobody connects it to the migration.

Acceptance Criteria

• A failure on the deprecated search path logs the exception class, message and stack trace, plus
  enough context to locate the call site — at minimum matching what the router path already emits.
• That log line contains at least one term an operator would plausibly grep for.
• The readiness report degrades rather than failing: an unreachable engine yields a report with
  that engine's side marked unavailable, the other side populated, and a verdict that reflects the
  partial view.
• ESIndexAPI.getIndicesStats() is not called unconditionally from the reconciler at phase 3.
• Regression test: stop the source engine at phase 3 and assert the endpoint returns a usable body.

Additional Context

Same lab run as #37635, which is the other way the readiness report stops telling the truth at phase 3
— there it answers, but with nothing in it. The two compound: after a phase-3 reindex the report goes
empty, and once the old cluster is off it stops answering entirely.