Reindex failures in dual-write phases do not report which engine failed

Problem Statement

In PHASE_1_DUAL_WRITE_ES_READS and PHASE_2_DUAL_WRITE_OS_READS every write fans out to two
engines, but reindex failure reporting is engine-agnostic. The UI shows only:

380 Content(s) failed to reindex and were skipped

There is no way to answer the questions an operator actually has:

• Did these fail on Elasticsearch, on OpenSearch, or on both?
• Are the same contentlets missing from both engines, or do the engines now disagree?
• Is the index I am about to read from (which depends on the phase) the one missing content?

GET /api/v1/esindex/reindex is the same — {"errorCount": 380, "inFullReindexation": false} — and
GET /api/v1/esindex/failed returns failureReason with no engine attribution.

This is not hypothetical. During a full reindex in phase 1, failures were 100 % asymmetric:

(Counts exceed 370 because of multiple language versions per identifier.)

The cause was a serialization limit reached only on the legacy
org.elasticsearch.client.RestHighLevelClient path and not on the OpenSearch 3.3.0 client path
(details in the companion issue on the ES bulk limit). The engine-specific nature of the failure was
invisible: the UI reported a single flat count, and total document counts actively concealed it —
the two working indices differed by only 45 documents overall while ES was missing 524 of these and
holding roughly 569 others that OpenSearch lacked.

The practical consequence: reads in phase 1 come from Elasticsearch, so those 370 contentlets were
unsearchable — while being perfectly present in OpenSearch. Switching to phase 2 would have made
them reappear, and switching back would have made them vanish again, with nothing in the UI
explaining why.

Diagnosing this required manual _count queries against both clusters plus cross-referencing the
indicies table. That should not be the only way to see it.

Related discoverability gaps found while diagnosing

• Failure counts are reported against no particular index generation. The affected contentlets
  existed in the previous ES generation (*_20260826165345) but not in the current one
  registered in indicies. Nothing indicated the current generation was incomplete.
• PhaseRouter logs shadow-write failures at a level set by DOTCMS_SHADOW_WRITE_LOG_LEVEL
  (default WARN), which is the right idea — but that signal never reaches the reindex
  failure surface the operator is actually looking at.

Steps to Reproduce

1. Configure DOT_FEATURE_FLAG_OPEN_SEARCH_PHASE=1 with distinct ES and OS endpoints.

2. Introduce a condition that fails on only one engine — e.g. a contentlet whose body
   exceeds 20 MB, which fails the ES bulk path but succeeds on the OS path (companion issue).

3. Run a full reindex.

4. Open Maintenance → Index. Observe N Content(s) failed to reindex and were skipped with no
   engine attribution.

5. Call GET /api/v1/esindex/reindex and GET /api/v1/esindex/failed — neither response
   names an engine.

6. Query both clusters directly to discover the failures are one-sided:

   # note: use identifier_dotraw, not identifier — the analyzed field splits UUIDs on dashes
   curl -sk -u <u>:<p> "https://<es>:9200/<working_index>/_search" \
     -H 'Content-Type: application/json' \
     -d '{"size":0,"query":{"terms":{"identifier_dotraw":["<id>", ...]}}}'
   
   curl -sk -u <u>:<p> "https://<os>:9200/<working_index>.os/_search" \
     -H 'Content-Type: application/json' \
     -d '{"size":0,"query":{"terms":{"identifier_dotraw":["<id>", ...]}}}'
   

7. Note that comparing total document counts between the two indices does not reveal the
   divergence, because failures in both directions offset each other.

Acceptance Criteria

• Reindex failure reporting attributes each failure to the engine(s) it failed on, in both
  the UI and GET /api/v1/esindex/failed.
• The UI distinguishes "failed on both engines" from "failed on one" — the latter means the
  engines have diverged and a phase switch will change search results.
• GET /api/v1/esindex/reindex reports per-engine error counts rather than a single
  errorCount.
• Divergence between write engines is detectable without manual cluster queries — e.g. a
  per-engine document count or an explicit mirror-consistency check on the current index
  generation. (SiteSearchAPI.writeMirrorsInSync() already does this for site-search
  indices; content indices have no equivalent.)
• Failure counts identify which index generation is incomplete.

dotCMS Version

dotcms/dotcms:trunk, opensearch.version 3.3.0. OpenSearch 1.3.20 (ES-compatible endpoint) +
OpenSearch 3.4.0, PHASE_1_DUAL_WRITE_ES_READS, PostgreSQL 16, ~1.55 M contentlets.

Severity

Medium - Some functionality impacted

Medium — no data is lost by this defect itself, but it hides the divergence caused by others,
which is what turns a recoverable error into an undiagnosed one.

Links

• ESIndexResource.java:180 — /failed, no engine attribution
• ESIndexResource.java:318 — getReindexationProgress, single flat errorCount
• PhaseRouter.java:251 — logShadowWriteFailure, the signal that never reaches the operator surface
• IndexConfigHelper.java:125 — MigrationPhase
• SiteSearchJobImpl.java:299 / writeMirrorsInSync() — the mirror-sync check that exists for
  site search but not for content indices

Freshdesk ticket: NA — found during internal ES→OpenSearch migration testing.