cockroachdb / cockroachdb/cockroach

jobs: logical replication metrics shared across tenants in shared-process multi-tenancy

Open
#171,977 4 comments 0 reactions 1 assignee Claimed by @andrew-r-thomas View on GitHub
A-disaster-recovery C-bug O-agent P-2 T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

The logical replication job's `metrics.Metrics` struct is built once at process `init()` in `pkg/crosscluster/logical/logical_replication_job.go` and registered via `jobs.WithJobMetrics(m)` (and `jobs.WithResolvedMetric(...)`, `jobs.WithJobClusterMetrics(...)`). `WithJobMetrics` stores that single pre-built instance in a process-global map and copies the same pointer into every SQL server's `jobs.Metrics.JobSpecificMetrics[TypeLogicalReplication]` ([pkg/jobs/metrics.go#L445](https://github.com/cockroachdb/cockroach/blob/b2948ddedacbd3c1b0c15ff7442501ac7d47f0f9/pkg/jobs/metrics.go#L445)).

In shared-process multi-tenancy, each SQL server gets its own `*metric.Registry` via `jobs.MakeRegistry` → `initMetrics`, but because the metric instances are shared, all tenants increment the same counters/gauges. Their LDR job metrics are conflated across tenants.

This is a latent correctness issue for shared-process multi-tenant deployments; it is harmless in dedicated-process deployments.

**Context**

The transactional LDR sub-metrics (coordinator/applier) were converted to the per-server `MakeXxxMetricsHook` pattern so each server constructs fresh instances. This issue tracks doing the same for the remaining legacy `metrics.Metrics`.

**Fix**

Convert the remaining `metrics.Metrics`, the cluster metrics passed to `WithJobClusterMetrics`, and the `ReplicatedTimeSeconds` gauge passed to `WithResolvedMetric` to the per-server hook pattern used by changefeed/backup/inspect (a `MakeXxxMetricsHook` invoked inside `initMetrics`), then drop the `init()`-time construction + `WithJobMetrics`/`WithResolvedMetric` path.

Jira issue: CRDB-65112

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.