cockroachdb / cockroachdb/cockroach
kvserver: noisy "descriptor changed" error logging
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
We frequently see lots of these errors in production clusters, in https://github.com/cockroachlabs/support/issues/2527 about 2500 in 12 hours:
```
unable to relocate range to []: while carrying out changes []:
change replicas of r failed: descriptor changed: [expected] ... != [actual] ...
```
These are typically entirely benign, when multiple actors (i.e. nodes/queues) are trying to execute replication conf changes on the same range simultaneously and have to retry (it's a failed cput), but it can cause undue concern with users.
We should either not log these, or only log them after a few retries have failed with the same error, and possibly downgrade the severity and/or soften the language. We have seen cases where these were caused by an actual bug though, and the conf changes never succeeded (#94834), so we probably don't want to remove the logging entirely.
Jira issue: CRDB-30534
Epic CRDB-58158
Contributor guide
Assessment
This issue has not been assessed yet.