cockroachdb / cockroachdb/cockroach
jobs: logical replication metrics shared across tenants in shared-process multi-tenancy
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
The logical replication job's `metrics.Metrics` struct is built once at process `init()` in `pkg/crosscluster/logical/logical_replication_job.go` and registered via `jobs.WithJobMetrics(m)` (and `jobs.WithResolvedMetric(...)`, `jobs.WithJobClusterMetrics(...)`). `WithJobMetrics` stores that single pre-built instance in a process-global map and copies the same pointer into every SQL server's `jobs.Metrics.JobSpecificMetrics[TypeLogicalReplication]` ([pkg/jobs/metrics.go#L445](https://github.com/cockroachdb/cockroach/blob/b2948ddedacbd3c1b0c15ff7442501ac7d47f0f9/pkg/jobs/metrics.go#L445)).
In shared-process multi-tenancy, each SQL server gets its own `*metric.Registry` via `jobs.MakeRegistry` → `initMetrics`, but because the metric instances are shared, all tenants increment the same counters/gauges. Their LDR job metrics are conflated across tenants.
This is a latent correctness issue for shared-process multi-tenant deployments; it is harmless in dedicated-process deployments.
**Context**
The transactional LDR sub-metrics (coordinator/applier) were converted to the per-server `MakeXxxMetricsHook` pattern so each server constructs fresh instances. This issue tracks doing the same for the remaining legacy `metrics.Metrics`.
**Fix**
Convert the remaining `metrics.Metrics`, the cluster metrics passed to `WithJobClusterMetrics`, and the `ReplicatedTimeSeconds` gauge passed to `WithResolvedMetric` to the per-server hook pattern used by changefeed/backup/inspect (a `MakeXxxMetricsHook` invoked inside `initMetrics`), then drop the `init()`-time construction + `WithJobMetrics`/`WithResolvedMetric` path.
Jira issue: CRDB-65112
Contributor guide
Assessment
This issue has not been assessed yet.