Sentry: replica_consistency.go:837: log.Fatal: ATTENTION: (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).computeChecksumPostApply.func2 | ...

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/6783109802/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/6783109802/?referrer=webhooks_plugin)

Panic Message:

```
replica_consistency.go:837: log.Fatal: ATTENTION:
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).computeChecksumPostApply.func2
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_consistency.go:837
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485
| runtime.goexit
| src/runtime/asm_arm64.s:1222
Wraps: (2) log.Fatal: ATTENTION:
|
| This node is terminating because a replica inconsistency was detected between [n16,s16,r7209/2:×]
| and its other replicas: (n13,s13):1,(n16,s16):2,(n20,s20):9. Please check your cluster-wide log files for more
| information and contact the CockroachDB support team. It is not necessarily safe
| to replace this node; cluster data may still be at risk of corruption.
|
| A checkpoints directory to aid (expert) debugging should be present in:
| /cockroach/cockroach-data/auxiliary
|
| A file preventing this node from restarting was placed at:
| /cockroach/cockroach-data/auxiliary/_CRITICAL_ALERT.txt
|
| Checkpoints are created on each node/store hosting this range, to help
| investigate the cause. Only nodes that are more likely to have incorrect data
| are terminated, and usually a majority of replicas continue running. Checkpoints
| are partial, i.e. contain only the data from to the inconsistent range, and
| possibly its neighbouring ranges.
|
| The storage checkpoint directories can/should be deleted when no longer needed.
| They are very helpful in debugging this issue, so before deleting them, please
| consider alternative actions:
|
| - If the store has enough capacity, hold off the deletion until CRDB staff has
| diagnosed the issue.
| - Back up the checkpoints for later investigation.
| - If the stores are nearly full, but the cluster has enough capacity, consider
| gradually decommissioning the affected nodes, to retain the checkpoints.
|
| To inspect the checkpoints, one can use the cockroach debug range-data tool, and
| command line tools like diff. For example:
|
| $ cockroach debug range-data --replicated data/auxiliary/checkpoints/rN_at_M N
|
| Note that a directory that ends with "_pending" might not represent a valid
| checkpoint. Such directories can exist if the node fails during checkpoint
| creation. These directories should be deleted, or inspected with caution.
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: ATTENTION:
replica_consistency.go:837: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_arm64.s#L1221-L1223](src/runtime/asm_arm64.s#L1221-L1223)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/stop/stopper.go#L484-L486](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/stop/stopper.go#L484-L486)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/replica_consistency.go#L836-L838](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/replica_consistency.go#L836-L838)

```
src/runtime/asm_arm64.s in runtime.goexit at line 1222
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 485
pkg/kv/kvserver/pkg/kv/kvserver/replica_consistency.go in pkg/kv/kvserver.(*Replica).computeChecksumPostApply.func2 at line 837
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v24.1.2 |
| Go Version | go1.22.3 X:nocoverageredesign |
| Platform | linux arm64 |
| Distribution | CCL |
| Cockroach Release | v24.1.2 |
| Cockroach SHA | 7e81be6de75205c3d08b0d8dcc6ca188306abc27 |
| # of CPUs | 12 |
| # of Goroutines | 577 |

Jira issue: CRDB-53106