OpenSearch: haltMigration() ES-fallback silently no-ops when the migration phase is system-table-sourced — startup proceeds ungated and search 500s while logs claim ES-only

Problem Statement

The emergency ES-only fallback (IndexConfigHelper.haltMigration() → MigrationPhase.reset() → Config.setProperty(FLAG_KEY, 0)) is a silent no-op when FEATURE_FLAG_OPEN_SEARCH_PHASE is sourced from the system table. Config.getIntProperty resolves env var → system table → props store (Config.java:634-643), but Config.setProperty writes only the props store — so after every "reset", MigrationPhase.current() still returns the migrating phase. MigrationPhase.reset()'s own javadoc documents this shadowing and tells callers to clear the system-table entry first, but no caller reads back or warns.

This defeats the phase-aware startup gate from #36244 and turns an OS outage into a functional search outage with actively misleading logs. Verified on dotcms/dotcms:nightly (rev f574a9de4a, 2026-07-09) with phase 2 seeded via system table, populated DB, OS3 stopped:

1. The boot log claims falling back to ES-only … Migration phase reset to PHASE_0 (was PHASE_2) four times in a single boot (MainServlet wait ×2, InitServlet validator, initIndex gate) — and each subsequent gate immediately logs phase=PHASE_2_DUAL_WRITE_OS_READS again.
2. MainServlet's re-wait is defeated: the second waitUtilIndexReady() is meant to gate the new primary (ES) after a shadow fallback; since the phase never reset, it re-gates the dead OS, fails, and startup proceeds with no primary store ever gated.
3. Runtime outage, not graceful fallback: POST /api/content/_search → HTTP 500 (Unable to load default versioned indices, falling back to default index via RuntimeExceptionMapper) — the phase-2 OS failure on the versioned-indices resolution path escapes PhaseRouter.read's per-read ES fallback.
4. The outage is masked to operators: GET /api/v1/esindex/cluster returns placeholder stats ({"clusterName":"opensearch"}) because OSIndexAPIImpl.getClusterStats() swallows connection failures (the same swallowing that made the pre-#36248 gate dead code).
5. The system-table key survives restarts, so every boot repeats the cycle.

Who it hits: deployments driving the phase via the system table — i.e. the restart-free runtime-flip path (the flow validated in QA-G17, and the natural way to walk a customer through phase changes mid-migration). Env-pinned deployments (docker/k8s) are unaffected: the halt works for the running JVM and merely reverts on restart.

Provenance: known-and-deferred, not missed — #36462's PR description says the system-table reset no-op is "worth validating explicitly in the migration QA scenario. Not changed by this PR." This is that validation; it upgraded the severity from "silent no-op" to "search outage misreported as graceful fallback".

Not a duplicate of:

• #36054 — same symptom family (versioned-indices resolution throws) but a different defect (Phase 3, unpopulated VersionedIndices); here the 500 is a consequence of the no-op leaving phase 2 pointed at a dead OS.
• The system-table non-integer phase parse defect noted in the G17 adversarial addendum (uncaught Integer.parseInt in Config) — a separate Config issue.

Steps to Reproduce

Rig: docker/docker-compose-examples/single-node-os-migration/, image dotcms/dotcms:nightly, with DOT_OS_CONNECTION_ATTEMPTS=2, DOT_OS_CONNECTION_RETRY_SLEEP_SECONDS=1 for fast iterations.

1. Start the stack without DOT_FEATURE_FLAG_OPEN_SEARCH_PHASE in the environment (compose !reset null) so the system table drives the phase. Let it boot at Phase 0.
2. Seed the phase via REST: POST /api/v1/system-table {"key":"FEATURE_FLAG_OPEN_SEARCH_PHASE","value":"2"}.
3. docker stop <opensearch3 container> and restart dotCMS.
4. Watch the boot log: repeated falling back to ES-only … (resetting FEATURE_FLAG_OPEN_SEARCH_PHASE to 0 via haltMigration) + Migration phase reset to PHASE_0… pairs, each followed by another gate logging phase=PHASE_2_DUAL_WRITE_OS_READS.
5. After boot: POST /api/content/_search → HTTP 500; GET /api/v1/esindex/cluster → {"clusterName":"opensearch"} (placeholder); GET /api/v1/system-table/FEATURE_FLAG_OPEN_SEARCH_PHASE → still "2".

Expected: the fallback actually takes effect (phase → 0, ES gated as the new primary, search served by ES) — or, if it cannot take effect, the log says so loudly instead of claiming success.

Acceptance Criteria

• After haltMigration() runs, MigrationPhase.current() actually returns PHASE_0_MIGRATION_NOT_STARTED when the phase was sourced from the system table (i.e. the halt clears/overrides the shadowing system-table key, per the reset() javadoc guidance).
• If the reset cannot take effect (e.g. env-pinned value re-asserts), the gate logs an explicit ERROR/FATAL stating the fallback did not apply and the phase remains unchanged — no success-claiming log on a no-op path.
• With the fix, the scenario above (system-table phase 2, OS down, restart) boots into working ES-only search: MainServlet's re-wait gates ES, POST /api/content/_search returns results, no repeated no-op halt loops.
• Phase-branching change covered by tests (unit for the read-back/clear logic; the existing OSIndexAPIImplWaitReadyIT pattern extended if practical).
• No behavior change for env/props-sourced phase deployments (current scenarios 4/6 behavior from the #36244 verification stays as-is).

dotCMS Version

main (f574a9de4a, dotcms/dotcms:nightly 2026-07-09); the affected haltMigration() callers shipped in #36248 (release 26.06.30-01) — but the underlying Config write/read asymmetry predates them.

Severity

Medium - Some functionality impacted

Links

• NA (no Freshdesk ticket)
• Found while verifying #36244 (see closing verification comment there); classification follow-up context in #36462.

Start with Config.java:634-643 and trace IndexConfigHelper.haltMigration() through MigrationPhase.reset() and MigrationPhase.current(). Compare the halt callers in MainServlet, InitServlet, and initIndex, then read OSIndexAPIImplWaitReadyIT. Done means system-table phases reset or fail loudly, ES-only startup and search work with OpenSearch unavailable, and env/props-sourced behavior remains unchanged.