dotCMS / dotCMS/core

feat(opensearch): observability — expose which backend (ES/OS) the PhaseRouter serves per index operation

Open
#36,340 0 comments 0 reactions 1 assignee View on GitHub

@fabrizzio-dotCMS is already working on this.

Since Jun 26, 2026.

Team : Scout
Dominant language
Java
Stars
970
Forks
486
Avg merge
3d 33m
Merged PRs (30d)
170

Description

Description

During the ES→OpenSearch migration, the routing decision (which backend serves each index read/write) is made internally by PhaseRouter and is not observable: a high-level API result is store-agnostic, so today neither operators (live migration) nor tests can tell which backend actually served a call. We only infer it from outcomes.

This proposes a small observability hook at the routing chokepoint, reusing existing infrastructure.

Where the decision is made (single chokepoint):

  • PhaseRouter<T> (dotCMS/src/main/java/com/dotcms/content/index/PhaseRouter.java): readProvider() (≈L144) and writeProviders() (≈L162–168), plus the dispatch wrappers read() / write() / writeBoolean() / writeReturning().
  • ⚠️ ESContentFactoryImpl does NOT use PhaseRouter — it routes inline via indexOperationsDelegate() (≈L271). The read/search path (searchHits, indexCount, indexSearchScroll) goes through there, so it must be instrumented too (or unified onto the router — see open question).

Reuse existing infra (don't invent):

  • Metrics: MetricsUtil.incrementCounter(name, tags...) / registerGauge(...) → Micrometer, auto-exported to Prometheus (/dotmgt/metrics) + JMX. Gated by metrics.enabled.
  • Backend identity: IndexTag (ES / .os) + MigrationPhase.current().

Proposed signals:

  • Counter dotcms.index.routing.selected{backend=ES|OS, operation=read|write|search|count, phase=0..3} — incremented on each routed op.
  • Gauge dotcms.index.migration.phase — current phase (live-migration dashboard).
  • (Optional) Timer dotcms.index.operation.duration{backend,operation,phase} for ES-vs-OS latency skew.

Test-assertability (second driver): add a lightweight routing trace (e.g. a ThreadLocal "last routed backend(s)" the router records, or an injectable recorder) so integration tests can assert "this high-level call was served by OpenSearch" — impossible today without bypassing the router. Complements MigrationPhaseStoreBootstrapIT (#36268), which can only assert presence/retrieval, not the routed path.

Acceptance Criteria

  • A Micrometer counter increments per routed index op, tagged backend / operation / phase, visible at /dotmgt/metrics (and JMX).
  • A gauge exposes the current migration phase.
  • Both routing surfaces covered: PhaseRouter and ESContentFactoryImpl.indexOperationsDelegate() (or the latter unified onto the router).
  • A test-facing trace/recorder lets an IT assert which backend served a high-level call; covered by a test.
  • Zero routing-behavior change; metrics respect metrics.enabled (no overhead when disabled).

Additional Context

Came out of PR #36268 / issue #36266 (phase-aware test bootstrap). Today tests can only assert outcome (content present/retrievable), not the path; this feature makes the routing decision observable for both QA and live-migration monitoring. Related: #36320.

Open question: instrument ESContentFactoryImpl in place, or unify its indexOperationsDelegate() onto PhaseRouter first (bigger, cleaner)? — to decide during implementation.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.