cockroachdb / cockroachdb/cockroach
spanconfig,kvserver: raciness when apply configs, resulting in stale configs
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
See this internal incident: https://github.com/cockroachlabs/support/issues/2491. It's possible for a range to persistedly have a stale span config through the following race condition. For MR tables, this could result in unexpected roundtrips/end-user latencies if replicas aren't in their configured zones.
1. Range splits due to size/load. RHS applies it through code path here. "Stale" config. Here:
https://github.com/cockroachdb/cockroach/blob/e96860479b6597a3ea4811ab21c3d5d3393901f8/pkg/kv/kvserver/replica_raftstorage.go#L403
2. We receive a spanconfig update that applies over both the RHS + LHS.
3. Individual stores learn about it and apply it to replicas with keyspans that overlap, so LHS + RHS replicas. Here: https://github.com/cockroachdb/cockroach/blob/f295bd861a3a427652b19c2254d2401ebb4a3c8e/pkg/kv/kvserver/store.go#L2433
But there's no explicit sequencing between steps 1 and 3. What prevents the stale config from being applied to the replica after the new config was applied? It's going to be left untouched after any config is applied, unless there's a further update.
Jira issue: CRDB-30322
Contributor guide
Assessment
This issue has not been assessed yet.