Sentry: replica_write.go:203: log.Fatal: replica is corrupted: replica corruption (processed=true): × (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(...

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/6853875859/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/6853875859/?referrer=webhooks_plugin)

Panic Message:

```
replica_write.go:203: log.Fatal: replica is corrupted: replica corruption (processed=true): ×
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go:203
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:512
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:186
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:203
| github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
| github.com/cockroachdb/cockroach/pkg/server/node.go:1436
| github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
| github.com/cockroachdb/cockroach/pkg/server/node.go:1631
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:704
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12
| github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
| github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:95
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:337
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:714
| github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8
| github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228
| github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:899
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:784
| github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:907
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:214
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:192
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2731
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2168
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatchAsync.func1
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2028
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485
| runtime.goexit
| src/runtime/asm_amd64.s:1695
Wraps: (2) secondary error attachment
| replica corruption (processed=true): ×
| (1) replica corruption (processed=true): ×
| Error types: (1) *kvpb.ReplicaCorruptionError
Wraps: (3) log.Fatal: replica is corrupted: replica corruption (processed=true): ×
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: replica is corrupted: replica corruption (processed=true): ×
*secondary.withSecondaryError
replica_write.go:203: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/stop/stopper.go#L484-L486](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/stop/stopper.go#L484-L486)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvclient/kvcoord/dist_sender.go#L2027-L2029](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvclient/kvcoord/dist_sender.go#L2027-L2029)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvclient/kvcoord/dist_sender.go#L2167-L2169](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvclient/kvcoord/dist_sender.go#L2167-L2169)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvclient/kvcoord/dist_sender.go#L2730-L2732](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvclient/kvcoord/dist_sender.go#L2730-L2732)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvclient/kvcoord/transport.go#L191-L193](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvclient/kvcoord/transport.go#L191-L193)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvclient/kvcoord/transport.go#L213-L215](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvclient/kvcoord/transport.go#L213-L215)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L906-L908](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L906-L908)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L783-L785](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L783-L785)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L898-L900](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L898-L900)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L227-L229](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L227-L229)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L713-L715](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L713-L715)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L814-L816](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L814-L816)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L133-L135](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L133-L135)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/stop/stopper.go#L336-L338](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/stop/stopper.go#L336-L338)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L135-L137](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L135-L137)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L814-L816](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L814-L816)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/auth.go#L94-L96](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/auth.go#L94-L96)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L814-L816](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L814-L816)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L168-L170](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L168-L170)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L814-L816](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L814-L816)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L96-L98](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L96-L98)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L703-L705](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L703-L705)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/server/node.go#L1630-L1632](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/server/node.go#L1630-L1632)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/server/node.go#L1435-L1437](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/server/node.go#L1435-L1437)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L202-L204](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L202-L204)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L192-L194](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L192-L194)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L185-L187](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L185-L187)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L511-L513](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L511-L513)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L202-L204](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L202-L204)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1695
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 485
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatchAsync.func1 at line 2028
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch at line 2168
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas at line 2731
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext at line 192
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch at line 214
pkg/rpc/pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 907
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func3 at line 784
pkg/rpc/pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 899
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewContext.ClientInterceptor.func8 at line 228
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 714
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 815
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 134
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 337
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 136
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 815
pkg/rpc/pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 95
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 815
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 169
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 815
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.ServerInterceptor.func12 at line 97
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 704
pkg/server/node.go in pkg/server.(*Node).Batch at line 1631
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1436
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 203
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 193
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 186
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 512
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).executeWriteBatch at line 203
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v24.1.2 |
| Go Version | go1.22.3 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v24.1.2 |
| Cockroach SHA | 7e81be6de75205c3d08b0d8dcc6ca188306abc27 |
| # of CPUs | 32 |
| # of Goroutines | 2005 |

Jira issue: CRDB-54100