cockroachdb / cockroachdb/cockroach

kvserver: noisy "descriptor changed" error logging

Open
#108,588 3 comments 0 reactions 0 assignees View on GitHub
A-kv-observability C-enhancement O-24.3-scale-testing O-support O-testcluster P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

We frequently see lots of these errors in production clusters, in https://github.com/cockroachlabs/support/issues/2527 about 2500 in 12 hours:

```
unable to relocate range to []: while carrying out changes []:
change replicas of r failed: descriptor changed: [expected] ... != [actual] ...
```

These are typically entirely benign, when multiple actors (i.e. nodes/queues) are trying to execute replication conf changes on the same range simultaneously and have to retry (it's a failed cput), but it can cause undue concern with users.

We should either not log these, or only log them after a few retries have failed with the same error, and possibly downgrade the severity and/or soften the language. We have seen cases where these were caused by an actual bug though, and the conf changes never succeeded (#94834), so we probably don't want to remove the logging entirely.

Jira issue: CRDB-30534

Epic CRDB-58158

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.