Sentry: replica_consistency.go:837: log.Fatal: ATTENTION: (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).computeChecksumPostApply.func2 | ...

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7314277126/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7314277126/?referrer=webhooks_plugin)

Panic Message:

```
replica_consistency.go:837: log.Fatal: ATTENTION:
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).computeChecksumPostApply.func2
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_consistency.go:837
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485
| runtime.goexit
| src/runtime/asm_amd64.s:1695
Wraps: (2) log.Fatal: ATTENTION:
|
| This node is terminating because a replica inconsistency was detected between [n1,s1,r1/1:×]
| and its other replicas: (n1,s1):1,(n2,s2):2,(n3,s3):3. Please check your cluster-wide log files for more
| information and contact the CockroachDB support team. It is not necessarily safe
| to replace this node; cluster data may still be at risk of corruption.
|
| A checkpoints directory to aid (expert) debugging should be present in:
| /cockroach/cockroach-data/auxiliary
|
| A file preventing this node from restarting was placed at:
| /cockroach/cockroach-data/auxiliary/_CRITICAL_ALERT.txt
|
| Checkpoints are created on each node/store hosting this range, to help
| investigate the cause. Only nodes that are more likely to have incorrect data
| are terminated, and usually a majority of replicas continue running. Checkpoints
| are partial, i.e. contain only the data from to the inconsistent range, and
| possibly its neighbouring ranges.
|
| The storage checkpoint directories can/should be deleted when no longer needed.
| They are very helpful in debugging this issue, so before deleting them, please
| consider alternative actions:
|
| - If the store has enough capacity, hold off the deletion until CRDB staff has
| diagnosed the issue.
| - Back up the checkpoints for later investigation.
| - If the stores are nearly full, but the cluster has enough capacity, consider
| gradually decommissioning the affected nodes, to retain the checkpoints.
|
| To inspect the checkpoints, one can use the cockroach debug range-data tool, and
| command line tools like diff. For example:
|
| $ cockroach debug range-data --replicated data/auxiliary/checkpoints/rN_at_M N
|
| Note that a directory that ends with "_pending" might not represent a valid
| checkpoint. Such directories can exist if the node fails during checkpoint
| creation. These directories should be deleted, or inspected with caution.
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: ATTENTION:
replica_consistency.go:837: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/util/stop/stopper.go#L484-L486](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/util/stop/stopper.go#L484-L486)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/replica_consistency.go#L836-L838](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/replica_consistency.go#L836-L838)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1695
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 485
pkg/kv/kvserver/pkg/kv/kvserver/replica_consistency.go in pkg/kv/kvserver.(*Replica).computeChecksumPostApply.func2 at line 837
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v24.1.3 |
| Go Version | go1.22.5 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v24.1.3 |
| Cockroach SHA | 9e10212477fde97f55ea4fff01797288c836575c |
| # of CPUs | 12 |
| # of Goroutines | 672 |

Jira issue: CRDB-61097