ua: system database in application tenant has lower replication factor than non-UA system database.

Open

#155,050 opened on Oct 8, 2025 1 comment 0 reactions 0 assignees View on GitHub

A-multitenancy branch-master C-bug T-db-server

Dominant language: Go
Stars: 32.5k
Forks: 4.1k
PR merge metrics: PR metrics pending

Description

**Describe the problem**

The system database in a non-UA cluster (and in the system cluster in a UA cluster) has a replication factor of 5:

```
demo@localhost:26257/system/defaultdb> show
target |
---------------------------------------------------+
RANGE default

DATABASE system

TABLE system.public.lease

RANGE meta

RANGE system

RANGE timeseries

RANGE liveness

TABLE system.public.replication_constraint_stats

TABLE system.public.replication_stats

TABLE system.public.statement_statistics

TABLE system.public.transaction_statistics

TABLE system.public.tenant_usage

TABLE system.public.span_stats_tenant_boundaries

TABLE system.public.statement_activity

TABLE system.public.transaction_activity

(15 rows) zone configurations; raw_config_sql ------------------------------------------------------------------------------ | ALTER RANGE default CONFIGURE ZONE USING | range_min_bytes = 134217728, | range_max_bytes = 536870912, | gc.ttlseconds = 14400, | num_replicas = 3, | constraints = '[]', | lease_preferences = '[]' | ALTER DATABASE system CONFIGURE ZONE USING | range_min_bytes = 134217728, | range_max_bytes = 536870912, | gc.ttlseconds = 14400, | num_replicas = 5, | constraints = '[]', | lease_preferences = '[]' | ALTER TABLE system.public.lease CONFIGURE ZONE USING | gc.ttlseconds = 600 | ALTER RANGE meta CONFIGURE ZONE USING | range_min_bytes = 134217728, | range_max_bytes = 536870912, | gc.ttlseconds = 3600, | num_replicas = 5, | constraints = '[]', | lease_preferences = '[]' | ALTER RANGE system CONFIGURE ZONE USING | range_min_bytes = 134217728, | range_max_bytes = 536870912, | gc.ttlseconds = 14400, | num_replicas = 5, | constraints = '[]', | lease_preferences = '[]' | ALTER RANGE timeseries CONFIGURE ZONE USING | gc.ttlseconds = 14400 | ALTER RANGE liveness CONFIGURE ZONE USING | range_min_bytes = 134217728, | range_max_bytes = 536870912, | gc.ttlseconds = 600, | num_replicas = 5, | constraints = '[]', | lease_preferences = '[]' | ALTER TABLE system.public.replication_constraint_stats CONFIGURE ZONE USING | gc.ttlseconds = 600 | ALTER TABLE system.public.replication_stats CONFIGURE ZONE USING | gc.ttlseconds = 600 | ALTER TABLE system.public.statement_statistics CONFIGURE ZONE USING | gc.ttlseconds = 3600 | ALTER TABLE system.public.transaction_statistics CONFIGURE ZONE USING | gc.ttlseconds = 3600 | ALTER TABLE system.public.tenant_usage CONFIGURE ZONE USING | gc.ttlseconds = 7200 | ALTER TABLE system.public.span_stats_tenant_boundaries CONFIGURE ZONE USING | gc.ttlseconds = 3600 | ALTER TABLE system.public.statement_activity CONFIGURE ZONE USING | gc.ttlseconds = 3600 | ALTER TABLE system.public.transaction_activity CONFIGURE ZONE USING | gc.ttlseconds = 3600

Time: 18ms total (execution 17ms / network 1ms)
```

In a UA-cluster's application tenant, however, no system database zone configuration is in place:

```
demo@127.0.0.1:26257/demoapp/movr> SHOW ZONE CONFIGURATIONS;
target | raw_config_sql
---------------------------------------------+------------------------------------------------------------------------
RANGE default | ALTER RANGE default CONFIGURE ZONE USING
| range_min_bytes = 134217728,
| range_max_bytes = 536870912,
| gc.ttlseconds = 14400,
| num_replicas = 3,
| constraints = '[]',
| lease_preferences = '[]'
TABLE system.public.statement_statistics | ALTER TABLE system.public.statement_statistics CONFIGURE ZONE USING
| gc.ttlseconds = 3600
TABLE system.public.transaction_statistics | ALTER TABLE system.public.transaction_statistics CONFIGURE ZONE USING
| gc.ttlseconds = 3600
TABLE system.public.statement_activity | ALTER TABLE system.public.statement_activity CONFIGURE ZONE USING
| gc.ttlseconds = 3600
TABLE system.public.transaction_activity | ALTER TABLE system.public.transaction_activity CONFIGURE ZONE USING
| gc.ttlseconds = 3600
(5 rows)

Time: 14ms total (execution 13ms / network 1ms)
```

Note also the difference in the gc.ttlseconds for the public.leases table.

This difference is also reflected in the final span configurations for the system tables:

NumReplicas According to Span Config

Time: 6ms total (execution 5ms / network 1ms)
```

We expect a UA cluster to provide users the same experience as a non-UA cluster, so these zone configuration differences are likely undersirable.

The core of the issue is here:

https://github.com/cockroachdb/cockroach/blob/54e9027656552387b727b1087896f7eae0509115/pkg/sql/catalog/bootstrap/metadata.go#L521

The initialization of both the system database and lease table zone configuration is below this early exit.

We may additionally consider an upgrade migration to fix this for existing clusters.

Jira issue: CRDB-55255

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

ua: system database in application tenant has lower replication factor than non-UA system database.

Description

Contributor guide

Assessment

Get new issues in your inbox