cockroachdb / cockroachdb/cockroach

spanconfig,kvserver: raciness when apply configs, resulting in stale configs

Open
#108,123 2 comments 0 reactions 0 assignees View on GitHub
A-zone-configs C-bug O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

See this internal incident: https://github.com/cockroachlabs/support/issues/2491. It's possible for a range to persistedly have a stale span config through the following race condition. For MR tables, this could result in unexpected roundtrips/end-user latencies if replicas aren't in their configured zones.

1. Range splits due to size/load. RHS applies it through code path here. "Stale" config. Here:
https://github.com/cockroachdb/cockroach/blob/e96860479b6597a3ea4811ab21c3d5d3393901f8/pkg/kv/kvserver/replica_raftstorage.go#L403

2. We receive a spanconfig update that applies over both the RHS + LHS.

3. Individual stores learn about it and apply it to replicas with keyspans that overlap, so LHS + RHS replicas. Here: https://github.com/cockroachdb/cockroach/blob/f295bd861a3a427652b19c2254d2401ebb4a3c8e/pkg/kv/kvserver/store.go#L2433

But there's no explicit sequencing between steps 1 and 3. What prevents the stale config from being applied to the replica after the new config was applied? It's going to be left untouched after any config is applied, unless there's a further update.

Jira issue: CRDB-30322

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.