feat(opensearch): observability — expose which backend (ES/OS) the PhaseRouter serves per index operation
@fabrizzio-dotCMS is already working on this.
Since Jun 26, 2026.
- Dominant language
- Java
- Stars
- 970
- Forks
- 486
- Avg merge
- 3d 33m
- Merged PRs (30d)
- 170
Description
Description
During the ES→OpenSearch migration, the routing decision (which backend serves each index read/write) is made internally by PhaseRouter and is not observable: a high-level API result is store-agnostic, so today neither operators (live migration) nor tests can tell which backend actually served a call. We only infer it from outcomes.
This proposes a small observability hook at the routing chokepoint, reusing existing infrastructure.
Where the decision is made (single chokepoint):
PhaseRouter<T>(dotCMS/src/main/java/com/dotcms/content/index/PhaseRouter.java):readProvider()(≈L144) andwriteProviders()(≈L162–168), plus the dispatch wrappersread()/write()/writeBoolean()/writeReturning().- ⚠️
ESContentFactoryImpldoes NOT usePhaseRouter— it routes inline viaindexOperationsDelegate()(≈L271). The read/search path (searchHits,indexCount,indexSearchScroll) goes through there, so it must be instrumented too (or unified onto the router — see open question).
Reuse existing infra (don't invent):
- Metrics:
MetricsUtil.incrementCounter(name, tags...)/registerGauge(...)→ Micrometer, auto-exported to Prometheus (/dotmgt/metrics) + JMX. Gated bymetrics.enabled. - Backend identity:
IndexTag(ES /.os) +MigrationPhase.current().
Proposed signals:
- Counter
dotcms.index.routing.selected{backend=ES|OS, operation=read|write|search|count, phase=0..3}— incremented on each routed op. - Gauge
dotcms.index.migration.phase— current phase (live-migration dashboard). - (Optional) Timer
dotcms.index.operation.duration{backend,operation,phase}for ES-vs-OS latency skew.
Test-assertability (second driver): add a lightweight routing trace (e.g. a ThreadLocal "last routed backend(s)" the router records, or an injectable recorder) so integration tests can assert "this high-level call was served by OpenSearch" — impossible today without bypassing the router. Complements MigrationPhaseStoreBootstrapIT (#36268), which can only assert presence/retrieval, not the routed path.
Acceptance Criteria
- A Micrometer counter increments per routed index op, tagged
backend/operation/phase, visible at/dotmgt/metrics(and JMX). - A gauge exposes the current migration phase.
- Both routing surfaces covered:
PhaseRouterandESContentFactoryImpl.indexOperationsDelegate()(or the latter unified onto the router). - A test-facing trace/recorder lets an IT assert which backend served a high-level call; covered by a test.
- Zero routing-behavior change; metrics respect
metrics.enabled(no overhead when disabled).
Additional Context
Came out of PR #36268 / issue #36266 (phase-aware test bootstrap). Today tests can only assert outcome (content present/retrievable), not the path; this feature makes the routing decision observable for both QA and live-migration monitoring. Related: #36320.
Open question: instrument ESContentFactoryImpl in place, or unify its indexOperationsDelegate() onto PhaseRouter first (bigger, cleaner)? — to decide during implementation.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.