OpenSearch Phase 1: a bad OpenSearch index permission stops dotCMS from creating any index on a new instance

Problem Statement

During the OpenSearch migration, Phase 1 is meant to keep Elasticsearch in charge of the site while OpenSearch runs alongside as a shadow copy. If OpenSearch is misconfigured, the site should not care.

It does. On a newly provisioned instance, an OpenSearch role that is not allowed to create dotCMS's indexes stops dotCMS from creating any index at all — including the Elasticsearch ones the site actually runs on. The instance comes up with no search index and cannot serve content.

	Expected in Phase 1	Actual
Elasticsearch index	created, instance works	not created ❌
OpenSearch index	not created, no pointer registered	not created ✅
building process in Admin → System → Index	visible	absent ❌
Migration phase	switches itself off to Phase 0	—
Log	one warning naming the rejected index pattern and how to fix it	no such warning ❌

Why this is a product bug, not a setup mistake

All of that behavior is already implemented and correct in the build tested (5ea351f), which is what makes this worth filing:

• Elasticsearch is created before OpenSearch is attempted — deliberately (ContentletIndexAPIImpl.bootstrapAndPoint:890-894, comment at :875-876).
• The OpenSearch 403 is caught and absorbed in Phase 1 (:1092 → handleOsBootstrapFailure:1142).
• The exact warning being asked for already exists — :1155-1178 logs likelyCause=AUTH_FORBIDDEN plus remediation naming the role's index pattern and DOT_DOTCMS_CLUSTER_ID.
• The two engines' index pointers are written independently, so there is no all-or-nothing commit (pointES:1214-1230 vs pointOS:1250-1269).
• Elasticsearch cannot be skipped on an empty database — esNeeded is true (initIndex:842-849).

The two obvious alternative explanations are both ruled out:

• "The phase could not be reset." Real hole, but not this: a phase set via the system table cannot be reset (IndexConfigHelper.java:187-193), whereas this was set with the DOT_FEATURE_FLAG_OPEN_SEARCH_PHASE environment variable, which resets correctly.
• "The build predates the fixes." 5ea351f contains #36783, #36903 and #36933 as verified ancestors.

Where the failure must be

Nothing in the Phase 1 OpenSearch path can suppress the Elasticsearch half. If no Elasticsearch index exists, execution never reached the code that creates one — the fault is upstream of everything previously fixed. On a new instance the first index creation runs on the starter-load path:

MainServlet.init → StartupTasksExecutor → Task00004LoadStarter
  → DotCMSInitDb.loadStarterSite:105 → refreshAllContent()
  → fullReindexStart() → initIndex() → bootstrapAndPoint(ts, true, true)
                                        ├─ bootstrapAndPointES   ← should always run
                                        └─ bootstrapAndPointOS   ← where the 403 happens

Most likely causes, in order:

1. The OpenSearch client is resolved before anyone checks the phase. ContentletIndexAPIImpl():199-201 and IndexAPIImpl():95-97 eagerly resolve the OpenSearch CDI beans in every phase. If the bean cannot be produced, APILocator.getContentletIndexAPI() throws and nothing is created in either engine, with nothing about index patterns in the log — matching the symptom exactly.
2. Startup aborts and rolls the Elasticsearch pointer back. fullReindexStart is @WrapInTransaction (:1367) and DotCMSInitDb.java:56 rethrows any starter-load failure. Two escape routes: pointOS (:1109) sits outside the OpenSearch try/catch, and bootstrapAndPointES catches only IOException (:1004-1008).
3. The starter load skipped index creation. refreshAllContent() returns early when isInFullReindex() is true (ESContentletAPIImpl:4049-4051) — e.g. if the starter import left rows in dist_reindex_journal. Unrelated to the phase.
4. The role also blocks cluster:monitor/main. A different path then runs first: a ~120 s retry loop (OSIndexAPIImpl:595-597) ending in not reachable after 24 attempt(s) rather than the index-pattern message. Elasticsearch is still created here, so it does not explain the missing index, but it is a real diagnostic trap and would explain a boot that looks stalled.

Also worth fixing while in here: ESIndexAPI.indexExists:359-363 calls indexName.toLowerCase() with no null guard, and on an empty database those names are null. The NPE is swallowed at :481-484 and happens to give the right answer — if it ever returned true, Elasticsearch would be skipped outright.

Why CI does not catch it

• The new-instance path is untested. bootstrapAndPoint( has zero test callers in the repo. OsBootstrapForbiddenIndexTest calls bootstrapAndPointOS, initAndPointReindex and fullReindexStart directly on an already-initialised fixture; it never boots from an empty database.
• Tests check bookkeeping, not the cluster. Assertions are on the database pointer (esAfter.getReindexWorking()), never on whether the index physically exists (esImpl().indexExists(...)). The only tests that do check the cluster run against a healthy OpenSearch.

Steps to Reproduce

Instance configuration Video: https://dotcms.slack.com/archives/C0B5XTNK0CV/p1786393580926099

Video screencasts:

https://github.com/user-attachments/assets/e5366ba5-cdb5-49a6-b56a-157d1f38b744

https://github.com/user-attachments/assets/5e5a419b-4c77-4c98-b6ce-3d5fafda6af6

An initial-setup walkthrough exists internally — ask @rjvelazco.

Order matters. Configure OpenSearch and the phase before the instance first boots. dotCMS writes the cluster id into dot_cluster on first boot and the stored value wins from then on, so a reused instance cannot reproduce this.

1. In OpenSearch, confirm the non-admin user and role exist and look correct in the admin panel — they should already be provisioned.
2. In Dashboards → Security → Roles → the role mapped to the dotCMS user → Index permissions, change the index pattern to one that cannot match dotCMS's index names, e.g. cluster_whatever_*. Leave the second block (pattern * with the monitor/stats actions) alone — this removes create and write in one move, which is what the scenario needs.
3. Set the migration phase to 1 using the DOT_FEATURE_FLAG_OPEN_SEARCH_PHASE environment variable. Do not use the system table — a phase set there cannot be reset (IndexConfigHelper.java:187-193).
4. Provision a fresh dotCMS instance with the correct cluster configuration. It must be a new instance with a clean database volume.
5. Once it boots, check Admin → System → Index and both search dashboards.

Result: no index in either engine, and no building index process in Admin → System → Index.

First triage step — these log lines identify the cause

Filter the boot log (unfiltered, INFO and above) for:

#	String	Source
1	Error creating index: ….os	OSIndexAPIImpl:234
2	OpenSearch index creation failed — …likelyCause=AUTH_FORBIDDEN	ContentletIndexAPIImpl:1173-1178
3	Migration phase reset to PHASE_0_MIGRATION_NOT_STARTED	IndexConfigHelper:201-205
4	Failed to create new indexes:	ContentletIndexAPIImpl:575
5	Building Initial Index	starter load

• 1–3 present → the OpenSearch failure was handled correctly; the missing Elasticsearch index is a separate problem downstream.
• All absent → startup aborted before index initialisation ran → cause 1, 2 or 3. This is what the symptom predicts.
• 2 present but reporting not reachable after 24 attempt(s) → cause 4.

Please attach the log as plain text rather than a share link, so the assignee can read it without extra access.

Acceptance Criteria

Behavior

• On a new instance in Phase 1, with an OpenSearch role whose index pattern cannot match dotCMS's index names, the Elasticsearch indexes are physically created and registered and the instance comes up working.
• The boot log carries one clear warning naming the cause and the fix. The message already exists at ContentletIndexAPIImpl:1155-1178 — confirm it is reached on this path.
• No OpenSearch index is created and no OpenSearch pointer is registered for indexes that do not exist.
• The migration halts itself to Phase 0; the Index screen lists only the Elasticsearch indexes.
• Publishing and searching work end to end, served by Elasticsearch.

Root cause

• Identify why bootstrapAndPointES is never reached, using the log strings above, and fix it.
• A failure resolving the OpenSearch CDI beans (ContentletIndexAPIImpl:199-201, IndexAPIImpl:95-97) can no longer block Elasticsearch index creation in phases 0–2.
• pointOS (:1109) can no longer roll back the already-committed pointES write inside the @WrapInTransaction fullReindexStart (:1367).
• bootstrapAndPointES's IOException-only catch (:1004-1008) no longer lets an unchecked exception abort the starter load.
• ESIndexAPI.indexExists:359-363 null-guards its argument.
• When the role also blocks cluster:monitor/main, the log points at the permission fix rather than only reporting the cluster unreachable.

Test coverage

• An integration test covers empty database, Phase 1, OpenSearch role that cannot create the indexes, end to end through bootstrapAndPoint(ts, true, true) — which has no test callers today.
• It asserts the Elasticsearch indexes physically exist via esImpl().indexExists(...), not just that a pointer was written, and that the warning is emitted.
• Registered in OpenSearchUpgradeSuite so it runs in CI.

Regression

• The same scenario behaves correctly when triggered by Reindex on an existing instance, not only on first boot.
• Restoring the index pattern and setting the phase back to 1 returns the instance to normal Phase 1 dual-write behavior.
• Phase 3 with the same misconfiguration still fails loudly and does not switch the phase off — there is nothing to fall back to.

dotCMS Version

dotcms-core 26.08.10-01 — August 10, 2026, buildNumber 5ea351f.

Confirmed to contain all three prior OpenSearch index-bootstrap fixes as ancestors: #36783 (merged Aug 1), #36903 (Aug 6), #36933 (Aug 7).

Environment: separate Elasticsearch and OpenSearch 3.x dev clusters, OpenSearch security plugin enabled, non-admin user and role.

Severity

High - Major functionality broken

Links

• NA — no Freshdesk ticket
• Prior QA history and the three fixes that came out of it: #36222, #36783, #36903, #36933
• Related: #36360 (OpenSearch site-search validation), #36244 (unify OpenSearch startup connection gate)