cockroachdb / cockroachdb/cockroach

Sentry: replica_write.go:198: log.Fatal: replica is corrupted: replica corruption (processed=true): × (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(...

Open
#159,212 1 comment 0 reactions 0 assignees View on GitHub
branch-release-25.3 C-bug O-sentry T-kv X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7102674032/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7102674032/?referrer=webhooks_plugin)

Panic Message:

```
replica_write.go:198: log.Fatal: replica is corrupted: replica corruption (processed=true): ×
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch
| pkg/kv/kvserver/replica_write.go:198
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
| pkg/kv/kvserver/replica_send.go:510
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
| pkg/kv/kvserver/replica_send.go:189
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
| pkg/kv/kvserver/store_send.go:188
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
| pkg/kv/kvserver/stores.go:199
| github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
| pkg/server/node.go:1654
| github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
| pkg/server/node.go:1847
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
| pkg/rpc/context.go:735
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12
| pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:96
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| pkg/rpc/context.go:846
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
| pkg/rpc/context.go:178
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| pkg/rpc/context.go:846
| github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
| pkg/rpc/auth.go:90
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| pkg/rpc/context.go:846
| github.com/cockroachdb/cockroach/pkg/rpc.gatewayRequestRecoveryInterceptor
| pkg/rpc/metrics.go:462
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| pkg/rpc/context.go:846
| github.com/cockroachdb/cockroach/pkg/server.newGRPCServer.NewRequestMetricsInterceptor.func4
| pkg/rpc/metrics.go:415
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| pkg/rpc/context.go:846
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
| pkg/rpc/context.go:136
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| pkg/util/stop/stopper.go:349
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
| pkg/rpc/context.go:134
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| pkg/rpc/context.go:846
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
| pkg/rpc/context.go:745
| github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8
| pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:236
| github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
| pkg/rpc/context.go:930
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3
| pkg/rpc/context.go:815
| github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch
| pkg/rpc/context.go:938
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch
| pkg/kv/kvclient/kvcoord/transport.go:211
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext
| pkg/kv/kvclient/kvcoord/transport.go:189
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas
| pkg/kv/kvclient/kvcoord/dist_sender.go:2779
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch
| pkg/kv/kvclient/kvcoord/dist_sender.go:2234
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges
| pkg/kv/kvclient/kvcoord/dist_sender.go:1772
Wraps: (2) secondary error attachment
| replica corruption (processed=true): ×
| (1) replica corruption (processed=true): ×
| Error types: (1) *kvpb.ReplicaCorruptionError
Wraps: (3) log.Fatal: replica is corrupted: replica corruption (processed=true): ×
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: replica is corrupted: replica corruption (processed=true): ×
*secondary.withSecondaryError
replica_write.go:198: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[pkg/kv/kvclient/kvcoord/dist_sender.go#L1771-L1773](pkg/kv/kvclient/kvcoord/dist_sender.go#L1771-L1773)
[pkg/kv/kvclient/kvcoord/dist_sender.go#L2233-L2235](pkg/kv/kvclient/kvcoord/dist_sender.go#L2233-L2235)
[pkg/kv/kvclient/kvcoord/dist_sender.go#L2778-L2780](pkg/kv/kvclient/kvcoord/dist_sender.go#L2778-L2780)
[pkg/kv/kvclient/kvcoord/transport.go#L188-L190](pkg/kv/kvclient/kvcoord/transport.go#L188-L190)
[pkg/kv/kvclient/kvcoord/transport.go#L210-L212](pkg/kv/kvclient/kvcoord/transport.go#L210-L212)
[pkg/rpc/context.go#L937-L939](pkg/rpc/context.go#L937-L939)
[pkg/rpc/context.go#L814-L816](pkg/rpc/context.go#L814-L816)
[pkg/rpc/context.go#L929-L931](pkg/rpc/context.go#L929-L931)
[pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L235-L237](pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L235-L237)
[pkg/rpc/context.go#L744-L746](pkg/rpc/context.go#L744-L746)
[pkg/rpc/context.go#L845-L847](pkg/rpc/context.go#L845-L847)
[pkg/rpc/context.go#L133-L135](pkg/rpc/context.go#L133-L135)
[pkg/util/stop/stopper.go#L348-L350](pkg/util/stop/stopper.go#L348-L350)
[pkg/rpc/context.go#L135-L137](pkg/rpc/context.go#L135-L137)
[pkg/rpc/context.go#L845-L847](pkg/rpc/context.go#L845-L847)
[pkg/rpc/metrics.go#L414-L416](pkg/rpc/metrics.go#L414-L416)
[pkg/rpc/context.go#L845-L847](pkg/rpc/context.go#L845-L847)
[pkg/rpc/metrics.go#L461-L463](pkg/rpc/metrics.go#L461-L463)
[pkg/rpc/context.go#L845-L847](pkg/rpc/context.go#L845-L847)
[pkg/rpc/auth.go#L89-L91](pkg/rpc/auth.go#L89-L91)
[pkg/rpc/context.go#L845-L847](pkg/rpc/context.go#L845-L847)
[pkg/rpc/context.go#L177-L179](pkg/rpc/context.go#L177-L179)
[pkg/rpc/context.go#L845-L847](pkg/rpc/context.go#L845-L847)
[pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L95-L97](pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L95-L97)
[pkg/rpc/context.go#L734-L736](pkg/rpc/context.go#L734-L736)
[pkg/server/node.go#L1846-L1848](pkg/server/node.go#L1846-L1848)
[pkg/server/node.go#L1653-L1655](pkg/server/node.go#L1653-L1655)
[pkg/kv/kvserver/stores.go#L198-L200](pkg/kv/kvserver/stores.go#L198-L200)
[pkg/kv/kvserver/store_send.go#L187-L189](pkg/kv/kvserver/store_send.go#L187-L189)
[pkg/kv/kvserver/replica_send.go#L188-L190](pkg/kv/kvserver/replica_send.go#L188-L190)
[pkg/kv/kvserver/replica_send.go#L509-L511](pkg/kv/kvserver/replica_send.go#L509-L511)
[pkg/kv/kvserver/replica_write.go#L197-L199](pkg/kv/kvserver/replica_write.go#L197-L199)

```
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges at line 1772
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch at line 2234
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas at line 2779
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext at line 189
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch at line 211
pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 938
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func3 at line 815
pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 930
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewContext.ClientInterceptor.func8 at line 236
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 745
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 846
pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 134
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 349
pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 136
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 846
pkg/rpc/metrics.go in pkg/server.newGRPCServer.NewRequestMetricsInterceptor.func4 at line 415
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 846
pkg/rpc/metrics.go in pkg/rpc.gatewayRequestRecoveryInterceptor at line 462
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 846
pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 90
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 846
pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 178
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 846
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.ServerInterceptor.func12 at line 96
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 735
pkg/server/node.go in pkg/server.(*Node).Batch at line 1847
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1654
pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 199
pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 188
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 189
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 510
pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).executeWriteBatch at line 198
```

### Tags

| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v25.3.2 |
| Go Version | go1.23.12 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.3.2 |
| Cockroach SHA | a5e060ee11ad11e1649e7569486b579e9da8c8d7 |
| # of CPUs | 16 |
| # of Goroutines | 713 |

Jira issue: CRDB-57730

Contributor guide

Open the contributing guide

Research direction

Start at pkg/kv/kvserver/replica_write.go:198 and trace the reported path through replica_send.go:510, using the Sentry report for context. A completed fix would need a confirmed root cause and regression coverage, but this issue does not identify either or provide a reproducible test.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.