Phase 2 ES read fallback never fires: content read path bypasses PhaseRouter, OpenSearch outage returns empty results

Problem Statement

The ES→OpenSearch migration design promises an automatic read fallback to Elasticsearch in Phase 2: if OpenSearch throws on a read, the error is logged at ERROR and the read is retried against ES, which is still active. That fallback exists and is correctly implemented in PhaseRouter, but the content read path never goes through PhaseRouter, so it never fires.

With OpenSearch unavailable in Phase 2 (ES up and healthy), POST /api/content/_search does not fall back. It returns either:

• HTTP 200 with resultsSize: 0 — a silent empty result. To a caller this is indistinguishable from "this content type has no content", so on a real site it renders as lost content with no error surfaced anywhere.
• HTTP 500 — depending on which catch branch the failure lands in.

Impact: any customer running Phase 2 loses the documented safety net. A transient OpenSearch outage, a slow node, or a missing OS index takes down content delivery instead of degrading to Elasticsearch. The silent-empty variant is the more dangerous of the two, because monitoring on 5xx will not catch it.

Found while validating Phase 2 on a real-data instance. Reproducible on demand.

Root cause

dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/ESContentFactoryImpl.java:273 selects the provider with a bare ternary:

ContentFactoryIndexOperations indexOperationsDelegate(){
    return isMigrationComplete() || isReadEnabled() ? indexOperationsOS : indexOperationsES ;
}

Its call sites — :1352 (search), :1607 (indexCount), :1616 (searchHits), :1634 (indexSearchScroll), :1669 (createScrollQuery) — invoke the chosen provider directly. PhaseRouter.read (PhaseRouter.java:187-199) and readChecked (:298-310) are never reached from this path.

The stack from the HTTP 500 shows the gap directly — there is no PhaseRouter frame:

com.dotmarketing.exception.DotRuntimeException: An error occurred when executing the
Lucene Query [ CountRequest@... ] : Connect to http://<os-host>:<port> failed: Connection refused
    at ContentFactoryIndexOperationsOS.cachedIndexCount(ContentFactoryIndexOperationsOS.java:192)
    at ContentFactoryIndexOperationsOS.indexCount(ContentFactoryIndexOperationsOS.java:429)
    at ESContentFactoryImpl.indexCount(ESContentFactoryImpl.java:1607)
    at ESContentletAPIImpl.indexCount(ESContentletAPIImpl.java:10553)
    at ContentletAPIInterceptor.indexCount(ContentletAPIInterceptor.java:3203)
    at ContentHelper.pullContent(ContentHelper.java:310)
    at ContentResource.search(ContentResource.java:251)

For contrast, the same outage on a path that does use the router (the health check) carries the router frame:

java.net.ConnectException: Connect to http://<os-host>:<port> failed: Connection refused
    at OSIndexAPIImpl.getClusterStats(OSIndexAPIImpl.java:522)
    at PhaseRouter.read(PhaseRouter.java:192)          <-- router frame present
    at IndexAPIImpl.getClusterStats(IndexAPIImpl.java:273)
    at ElasticsearchHealthCheck.testElasticsearchConnectivity(ElasticsearchHealthCheck.java:58)

Second layer — why the same outage yields 200/total=0 for some types and 500 for others

Corrected 2026-09-07. An earlier revision of this issue attributed the 200 / total=0
responses to the OpenSearchException → ERROR_HIT branch in
ContentFactoryIndexOperationsOS. That is wrong: a ConnectException is not an
OpenSearchException, so that branch is never reached in the case reported here. The real
mechanism is below, verified against the tree at the commit named in dotCMS Version.

A connection failure lands in the generic catch (Exception) branch
(ContentFactoryIndexOperationsOS.java:126-130), which throws DotRuntimeException. From
there, three things decide what the caller sees:

1. ContentHelper.java:310 runs the count first. A CountRequest carries no offset/limit,
   so varying the offset does not change the count cache key. Content types queried before
   the outage had a cached count and did not throw. Types not previously queried (Hero,
   ContactCard) missed the cache, the count threw → 500.
2. ContentUtils.java:302 swallows the search failure. A catch (Throwable) logs a
   one-line-truncated WARN and returns an empty list.
3. ContentHelper.java:318 erases the count.
   if (contentlets.isEmpty() && offset <= resultsSize) { resultsSize = 0; } overwrites the
   real cached count with 0 → 200 / total=0.

So the split is not two different exception branches — it is one failure, plus whether the
count happened to be cached.

Note that (2) and (3) are pre-existing and phase-independent: they behave identically with
Elasticsearch as the read engine in Phase 0. They explain the shape of the symptom, not the
defect. Fixing the missing routing removes the Phase 2 case; changing them repo-wide is a
separate, larger concern and should be filed on its own.

Second-order hazard — ERROR_HIT still needs handling

Independently of the above, the OpenSearchException branch
(ContentFactoryIndexOperationsOS.java:109-121) does convert a genuine OpenSearch failure into
a legitimate-looking empty result:

} catch (final OpenSearchException e) {
    ... Logger.warn(...) ...
    if (shouldQueryCache(exceptionMsg)) { queryCache.put(searchRequest, ERROR_HIT); }
    return ERROR_HIT;          // :51-54 — hits[], total=0, maxScore=0
}

This matters for failures that are OpenSearchException — a missing OpenSearch counterpart
index, a bad mapping, a parse error. In those cases even a correctly wired router has no
exception to catch, so the fallback still would not fire.

This ERROR_HIT pattern is inherited Elasticsearch behaviour, not new to the migration —
ContentFactoryIndexOperationsES.java:56 and :151-161 are structurally identical. Any change
to it must first enumerate the existing callers that depend on receiving an empty result rather
than an exception.

Also observed

• Failures are logged at WARN, not the documented ERROR, so the "early-warning signal per read" described in the design is weaker than specified.
• docs/backend/OPENSEARCH_MIGRATION.md:402 states that in Phase 2 a missing OS counterpart index is caught by the read fallback. That protection does not exist either, which affects the reactivated-backup-index scenario.
• Cache poisoning is bounded: shouldQueryCache(exceptionMsg) (:78-88) only caches ERROR_HIT for parse_exception / search_phase_execution_exception, so a connection failure is not cached.

Design being violated

docs/backend/OPENSEARCH_MIGRATION.md

Line	Text
105	"Read fallback (Phase 2 only) — In Phase 2 OS serves reads but ES is still active. If OS throws an exception on a read..."
901	"OS read failures fall back to ES in Phase 2 — PhaseRouter catches the exception, logs at ERROR, and retries against ES"
676	Phase 2: "Still works: the Phase-2 read fallback drops back to Elasticsearch, but logs an ERROR per read — the early-warning signal"
402	"In Phase 2 a missing OS twin is caught by the read fallback (OS errors → read from ES)"

Steps to Reproduce

Precondition: an instance in Phase 2 (DOT_FEATURE_FLAG_OPEN_SEARCH_PHASE=2), Elasticsearch up and healthy, OpenSearch up, content indexed and in sync on both engines.

1. Confirm the live phase from GET /api/v1/jvm — DOT_FEATURE_FLAG_OPEN_SEARCH_PHASE must report 2. Confirm GET /api/v1/index/migration/readiness reports readEngine: OpenSearch.

2. Stop the OpenSearch node (container stop, or block its port). Leave Elasticsearch running.

3. Call POST /api/content/_search for several content types that do have live content:

   {"query":"+contentType:Profile +live:true","limit":7,"offset":1}
   

   Vary the offset on every call. An identical request body is served from the query cache and returns the pre-outage result, which looks like a successful fallback and hides the bug entirely. This is how the defect was nearly missed during QA.

4. Observe the responses.

5. Restart OpenSearch and repeat the same calls to confirm the content was there all along.

Expected vs actual

Content type	Expected (per design)	Actual with OS down	Same call, OS up
Profile	results served by ES, ERROR logged	200, total=0	200, total=184
Testimonials	results served by ES, ERROR logged	200, total=0	200, total=317
JobPosting	results served by ES, ERROR logged	200, total=0	200, total=27
Hero	results served by ES, ERROR logged	500	200, total=215
ContactCard	results served by ES, ERROR logged	500	200, total=106

Elasticsearch held complete, in-sync copies of all of it for the whole duration of the outage.

Acceptance Criteria

• All five read call sites of ESContentFactoryImpl.indexOperationsDelegate() (:1352, :1607, :1616, :1634, :1669) route through PhaseRouter.read / readChecked rather than invoking the selected provider directly.
• In Phase 2, with OpenSearch unavailable, POST /api/content/_search returns the same non-zero result set that Elasticsearch holds — no total=0, no 500 — for every content type that has live content.
• Each fallback occurrence is logged at ERROR (not WARN), naming the failing OS operation and the cause, so an outage is visible in monitoring per the design's "early-warning signal".
• For failures that are an OpenSearchException — a missing OpenSearch counterpart index, a bad mapping, a parse error — the branch at ContentFactoryIndexOperationsOS.java:109-121 no longer converts the failure into a valid empty result while in Phase 2, so the router has an exception to act on. (This branch is not what produced the 200 / total=0 observed above — see Second layer — so it can ship separately from the routing fix if the caller enumeration below turns out to be large.)
• Changing that branch does not alter behaviour for Elasticsearch (ContentFactoryIndexOperationsES.java:151-161) nor for Phase 3, where the design specifies no fallback and failures must propagate. Callers that currently rely on receiving an empty result instead of an exception are enumerated and confirmed unaffected.
• Phase 0, 1 and 3 read behaviour is unchanged: 0/1 read from ES with no fallback needed, 3 reads from OS and propagates failures.
• A missing OpenSearch counterpart index in Phase 2 is served from Elasticsearch rather than returning empty, matching OPENSEARCH_MIGRATION.md:402.
• Integration test: with the OS provider stubbed to throw, a Phase-2 content search returns the ES result set and logs at ERROR. Registered in the matching @SuiteClasses suite so it actually runs in CI.
• Integration test: the same stubbed failure in Phase 3 propagates and does not fall back to ES.
• Regression test covering the cache path: a repeated identical query must not mask a live OS failure by serving a stale pre-outage result.

dotCMS Version

main branch, commit 788795e915 (2026-09-03). Reproduced on a local instance running Phase 2 of the ES→OpenSearch migration, with Elasticsearch 7.10.2 and OpenSearch 3.4.0.

Severity

High - Major functionality broken

Links

• Spec (Spec-Kit PR 1, spec only — no implementation): #37438
• specs/37413-phase2-read-fallback/spec.md