cockroachdb / cockroachdb/cockroach

Sentry: replica_consistency.go:822: log.Fatal: ATTENTION: (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).computeChecksumPostApply.func2 | ...

Open
#158,356 1 comment 0 reactions 0 assignees View on GitHub
branch-release-25.3 C-bug O-sentry T-kv X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7064476424/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7064476424/?referrer=webhooks_plugin)

Panic Message:

```
replica_consistency.go:822: log.Fatal: ATTENTION:
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).computeChecksumPostApply.func2
| pkg/kv/kvserver/replica_consistency.go:822
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1
| pkg/util/stop/stopper.go:438
| runtime.goexit
| src/runtime/asm_amd64.s:1700
Wraps: (2) log.Fatal: ATTENTION:
|
| This node is terminating because a replica inconsistency was detected between [n2,s2,r12/3:/Table/{8-9}]
| and its other replicas: (n1,s1):1,(n3,s3):2,(n2,s2):3. Please check your cluster-wide log files for more
| information and contact the CockroachDB support team. It is not necessarily safe
| to replace this node; cluster data may still be at risk of corruption.
|
| A checkpoints directory to aid (expert) debugging should be present in:
| /cockroach/cockroach-data/auxiliary
|
| A file preventing this node from restarting was placed at:
| /cockroach/cockroach-data/auxiliary/_CRITICAL_ALERT.txt
|
| Checkpoints are created on each node/store hosting this range, to help
| investigate the cause. Only nodes that are more likely to have incorrect data
| are terminated, and usually a majority of replicas continue running. Checkpoints
| are partial, i.e. contain only the data from to the inconsistent range, and
| possibly its neighbouring ranges.
|
| The storage checkpoint directories can/should be deleted when no longer needed.
| They are very helpful in debugging this issue, so before deleting them, please
| consider alternative actions:
|
| - If the store has enough capacity, hold off the deletion until CRDB staff has
| diagnosed the issue.
| - Back up the checkpoints for later investigation.
| - If the stores are nearly full, but the cluster has enough capacity, consider
| gradually decommissioning the affected nodes, to retain the checkpoints.
|
| To inspect the checkpoints, one can use the cockroach debug range-data tool, and
| command line tools like diff. For example:
|
| $ cockroach debug range-data --replicated data/auxiliary/checkpoints/rN_at_M N
|
| Note that a directory that ends with "_pending" might not represent a valid
| checkpoint. Such directories can exist if the node fails during checkpoint
| creation. These directories should be deleted, or inspected with caution.
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: ATTENTION:
replica_consistency.go:822: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1699-L1701](src/runtime/asm_amd64.s#L1699-L1701)
[pkg/util/stop/stopper.go#L437-L439](pkg/util/stop/stopper.go#L437-L439)
[pkg/kv/kvserver/replica_consistency.go#L821-L823](pkg/kv/kvserver/replica_consistency.go#L821-L823)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1700
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1 at line 438
pkg/kv/kvserver/replica_consistency.go in pkg/kv/kvserver.(*Replica).computeChecksumPostApply.func2 at line 822
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v25.3.4 |
| Go Version | go1.23.12 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.3.4 |
| Cockroach SHA | 1cddb3a22695e967197844b214115eb645473efe |
| # of CPUs | 4 |
| # of Goroutines | 474 |

Jira issue: CRDB-57252

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.