Sentry: replica_write.go:198: log.Fatal: replica is corrupted: replica corruption (processed=true): × (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(...

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/6807712547/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/6807712547/?referrer=webhooks_plugin)

Panic Message:

```
replica_write.go:198: log.Fatal: replica is corrupted: replica corruption (processed=true): ×
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch
| pkg/kv/kvserver/replica_write.go:198
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
| pkg/kv/kvserver/replica_send.go:498
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
| pkg/kv/kvserver/replica_send.go:177
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
| pkg/kv/kvserver/store_send.go:188
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
| pkg/kv/kvserver/stores.go:197
| github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
| pkg/server/node.go:1641
| github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
| pkg/server/node.go:1842
| github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler.func1
| bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10810
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12
| pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:92
| google.golang.org/grpc.getChainUnaryHandler.func1
| external/org_golang_google_grpc/server.go:1154
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
| pkg/rpc/context.go:165
| google.golang.org/grpc.getChainUnaryHandler.func1
| external/org_golang_google_grpc/server.go:1154
| github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
| pkg/rpc/auth.go:90
| google.golang.org/grpc.getChainUnaryHandler.func1
| external/org_golang_google_grpc/server.go:1154
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
| pkg/rpc/context.go:132
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| pkg/util/stop/stopper.go:349
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
| pkg/rpc/context.go:130
| google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1
| external/org_golang_google_grpc/server.go:1145
| github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler
| bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10812
| google.golang.org/grpc.(*Server).processUnaryRPC
| external/org_golang_google_grpc/server.go:1335
| google.golang.org/grpc.(*Server).handleStream
| external/org_golang_google_grpc/server.go:1712
| google.golang.org/grpc.(*Server).serveStreams.func1.1
| external/org_golang_google_grpc/server.go:947
| runtime.goexit
| src/runtime/asm_arm64.s:1222
Wraps: (2) secondary error attachment
| replica corruption (processed=true): ×
| (1) replica corruption (processed=true): ×
| Error types: (1) *kvpb.ReplicaCorruptionError
Wraps: (3) log.Fatal: replica is corrupted: replica corruption (processed=true): ×
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: replica is corrupted: replica corruption (processed=true): ×
*secondary.withSecondaryError
replica_write.go:198: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_arm64.s#L1221-L1223](src/runtime/asm_arm64.s#L1221-L1223)
[external/org_golang_google_grpc/server.go#L946-L948](external/org_golang_google_grpc/server.go#L946-L948)
[external/org_golang_google_grpc/server.go#L1711-L1713](external/org_golang_google_grpc/server.go#L1711-L1713)
[external/org_golang_google_grpc/server.go#L1334-L1336](external/org_golang_google_grpc/server.go#L1334-L1336)
[bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/https://github.com/cockroachdb/cockroach/blob/b33c70a891bb565033304c0411af9875506051f3/pkg/kv/kvpb/api.pb.go#L10811-L10813](bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/https://github.com/cockroachdb/cockroach/blob/b33c70a891bb565033304c0411af9875506051f3/pkg/kv/kvpb/api.pb.go#L10811-L10813)
[external/org_golang_google_grpc/server.go#L1144-L1146](external/org_golang_google_grpc/server.go#L1144-L1146)
[pkg/rpc/context.go#L129-L131](pkg/rpc/context.go#L129-L131)
[pkg/util/stop/stopper.go#L348-L350](pkg/util/stop/stopper.go#L348-L350)
[pkg/rpc/context.go#L131-L133](pkg/rpc/context.go#L131-L133)
[external/org_golang_google_grpc/server.go#L1153-L1155](external/org_golang_google_grpc/server.go#L1153-L1155)
[pkg/rpc/auth.go#L89-L91](pkg/rpc/auth.go#L89-L91)
[external/org_golang_google_grpc/server.go#L1153-L1155](external/org_golang_google_grpc/server.go#L1153-L1155)
[pkg/rpc/context.go#L164-L166](pkg/rpc/context.go#L164-L166)
[external/org_golang_google_grpc/server.go#L1153-L1155](external/org_golang_google_grpc/server.go#L1153-L1155)
[pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L91-L93](pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L91-L93)
[bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/https://github.com/cockroachdb/cockroach/blob/b33c70a891bb565033304c0411af9875506051f3/pkg/kv/kvpb/api.pb.go#L10809-L10811](bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/https://github.com/cockroachdb/cockroach/blob/b33c70a891bb565033304c0411af9875506051f3/pkg/kv/kvpb/api.pb.go#L10809-L10811)
[pkg/server/node.go#L1841-L1843](pkg/server/node.go#L1841-L1843)
[pkg/server/node.go#L1640-L1642](pkg/server/node.go#L1640-L1642)
[pkg/kv/kvserver/stores.go#L196-L198](pkg/kv/kvserver/stores.go#L196-L198)
[pkg/kv/kvserver/store_send.go#L187-L189](pkg/kv/kvserver/store_send.go#L187-L189)
[pkg/kv/kvserver/replica_send.go#L176-L178](pkg/kv/kvserver/replica_send.go#L176-L178)
[pkg/kv/kvserver/replica_send.go#L497-L499](pkg/kv/kvserver/replica_send.go#L497-L499)
[pkg/kv/kvserver/replica_write.go#L197-L199](pkg/kv/kvserver/replica_write.go#L197-L199)

```
src/runtime/asm_arm64.s in runtime.goexit at line 1222
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).serveStreams.func1.1 at line 947
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).handleStream at line 1712
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).processUnaryRPC at line 1335
bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go in pkg/kv/kvpb._Internal_Batch_Handler at line 10812
external/org_golang_google_grpc/server.go in google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1 at line 1145
pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 130
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 349
pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 132
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainUnaryHandler.func1 at line 1154
pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 90
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainUnaryHandler.func1 at line 1154
pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 165
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainUnaryHandler.func1 at line 1154
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.ServerInterceptor.func12 at line 92
bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go in pkg/kv/kvpb._Internal_Batch_Handler.func1 at line 10810
pkg/server/node.go in pkg/server.(*Node).Batch at line 1842
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1641
pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 197
pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 188
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 177
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 498
pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).executeWriteBatch at line 198
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v24.3.15 |
| Go Version | go1.22.8 X:nocoverageredesign |
| Platform | linux arm64 |
| Distribution | CCL |
| Cockroach Release | v24.3.15 |
| Cockroach SHA | b33c70a891bb565033304c0411af9875506051f3 |
| # of CPUs | 32 |
| # of Goroutines | 9792 |

Jira issue: CRDB-53474