cockroachdb / cockroachdb/cockroach
kvserver: RequestCancelled error during AdminSplit is reported as replica corruption
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
While running
```
RESTORE DATABASE tpce FROM LATEST IN 's3://cockroach-fixtures-us-east-2/backups/tpc-e/customers=25000/v23.1.11/inc-count=48/rev-history=false?AUTH=implicit' AS OF SYSTEM TIME '2023-11-08T20:00:44.127379Z' WITH EXPERIMENTAL DEFERRED COPY, unsafe_restore_incompatible_version;
```
On an empty cluster over a slow network link, I observed the following node crash:
```
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 replica is corrupted: replica corruption (processed=true): could not read from AbortSpan: failed to get s3 object: RequestCanceled: RequestCanceled: request context canceled
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 +caused by: context canceled
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !goroutine 13807 [running]:
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !runtime/debug.Stack()
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! GOROOT/src/runtime/debug/stack.go:24 +0x64
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x14001eca8d0, 0x24}, {0x1091c984d, 0x1}, {0x1091c984d, 0x1}, {0x1090f86e4, 0x6}, {0x1091c984d, ...}}, ...})
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/util/log/clog.go:280 +0xc4
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x10b8abdb0, 0x14004a825d0}, 0x3, 0x4, 0x0, 0x0?, {0x109167074, 0x18}, {0x1400ff958d0, 0x1, ...})
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/util/log/channels.go:109 +0x454
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(...)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/util/log/channels.go:39
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.FatalfDepth(...)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/bazel-out/darwin_arm64-fastbuild/bin/pkg/util/log/log_channels_generated.go:920
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).setCorruptRaftMuLocked(0x14013ca1208, {0x10b8abdb0, 0x14004a825d0}, 0x1400fc7e9c0)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_corruption.go:71 +0x33c
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch(0x14013ca1208, {0x10b8abdb0, 0x14004a82360}, 0x1400b60aa20, 0x14024c3c680)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go:203 +0xd48
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0x14013ca1208, {0x10b8abdb0, 0x14004a82360}, 0x1400b60aa20, 0x10b835d90)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:512 +0x234
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0x14013ca1208, {0x10b8abdb0?, 0x14004a822a0?}, 0x1400b60aa20)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:186 +0x4e8
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0x14001b5e008, {0x10b8abdb0?, 0x14004a82270?}, 0x1400b60aa20)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193 +0x5a4
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0x140013c1e00, {0x10b8abdb0, 0x14004a82270}, 0x1400b60aa20)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:203 +0x60
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0x14002104808, {0x10b8abdb0?, 0x14004a82240?}, {0x1400261db00?}, 0x1400b60aa20)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/server/node.go:1414 +0x374
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0x14002104808, {0x10b8abdb0, 0x14004a82120}, 0x1400b60aa20)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/server/node.go:1548 +0x224
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1({0x10b8abdb0?, 0x14004a82120?}, {0x10b7bf3a0?, 0x1400b60aa20?})
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:704 +0x54
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0x10b8abdb0, 0x14004a82120}, {0x10b7bf3a0, 0x1400b60aa20}, 0x14003f7e660, 0x14003f25da0)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97 +0x32c
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x10b8abdb0?, 0x14004a82120?}, {0x10b7bf3a0?, 0x1400b60aa20?})
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x3c
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0x10b8abdb0, 0x14004a82120}, {0x10b7bf3a0, 0x1400b60aa20}, 0x14002580000?, 0x14003f7e680)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169 +0x88
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x10b8abdb0?, 0x14004a82120?}, {0x10b7bf3a0?, 0x1400b60aa20?})
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x3c
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor({0x14002580000?, {{0x104795d50?}, {0x10b8ea880?, 0x140025a8840?}}}, {0x10b8abdb0, 0x14004a82120}, {0x10b7bf3a0, 0x1400b60aa20}, 0x14003f7e660, 0x14003f7e6a0)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:95 +0x15c
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x10b8abdb0?, 0x14004a82120?}, {0x10b7bf3a0?, 0x1400b60aa20?})
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x3c
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0x10b8abdb0?, 0x14004a82120?})
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136 +0x40
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0x140019bb320, {0x10b8abdb0, 0x14004a82120}, {0x140087717a0?, 0x1400ff983c8?}, 0x1400ff980b0)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336 +0xac
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0x10b8abdb0?, 0x14004a82120?}, {0x10b7bf3a0?, 0x1400b60aa20?}, 0x1400ff98138?, 0x105e2b0ac?)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134 +0x7c
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x10b8abdb0?, 0x14004a82120?}, {0x10b7bf3a0?, 0x1400b60aa20?})
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x3c
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2({0x10b8abdb0?, 0x14004a82120?}, {0x1400ff981e8?, 0x104795d50?}, {0x10b7bf3a0?, 0x1400b60aa20?}, {0x10b643e60, 0x1400438fa80}, 0x1400ff981e8?, {0x104795828?, ...})
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:714 +0x44
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8({0x10b8abdb0, 0x14004a82120}, {0x1091aec1a, 0x21}, {0x10b7bf3a0, 0x1400b60aa20}, {0x10b643e60, 0x1400438fa80}, 0x0, 0x140020fd040, ...)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228 +0x2e8
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1({0x10b8abdb0, 0x14004a82120}, {0x1091aec1a, 0x21}, {0x10b7bf3a0, 0x1400b60aa20}, {0x10b643e60, 0x1400438fa80}, 0x0, {0x0, ...})
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:899 +0xdc
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3({0x10b8abd78, 0x1400b42efa0}, 0x1400b60a900, {0x0, 0x0, 0x0})
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:784 +0x2a0
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:907
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch(0x1401bea36e0, {0x10b8abd78, 0x1400b42efa0}, 0x1e405bf0?, {0x10b89bf60, 0x14003fc3500}, 0x1400b60a900)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:214 +0x14c
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext(0x1401bea36e0, {0x10b8abd78, 0x1400b42efa0}, 0x1400b60a900)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:192 +0x90
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(0x1400156c688, {0x10b8abdb0, 0x14004a820c0}, 0x1400b60a5a0?, {0x140010f5950, 0x14014466d00}, 0x0)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2473 +0xcb0
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(0x1400156c688, {0x10b8abdb0, 0x14004a820c0}, 0x1400b60a5a0, {{0x1400a9b9713, 0x1b, 0x21}, {0x1400a9b9713, 0x1c, 0x21}}, ...)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1946 +0x518
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(0x1400156c688, {0x10b8abdb0, 0x14004a820c0}, 0x1400b60a5a0, {{0x1400a9b9713, 0x1b, 0x21}, {0x1400a9b9713, 0x1c, 0x21}}, ...)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1498 +0x25c
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(0x1400156c688, {0x10b8abdb0, 0x14004a82030}, 0x1400b60a5a0)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1114 +0x4b0
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked(0x1401bcb75c8, {0x10b8abdb0, 0x14004a82030}, 0x1400b60a5a0)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:82 +0x188
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked(0x1401bcb7578, {0x10b8abdb0?, 0x14004a82030?}, 0x0?)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go:46 +0xc4
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts(0x1401bcb7498, {0x10b8abdb0, 0x14004a82030}, 0x1400b60a5a0, 0x5)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:239 +0x148
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked(0x1401bcb7498, {0x10b8abdb0, 0x14004a82030}, 0x0?)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:167 +0xac
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked(0x1401bcb7458, {0x10b8abdb0, 0x14004a82030}, 0x1400b60a5a0)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go:149 +0x410
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked(0x1401bcb7328, {0x10b8abdb0, 0x14004a82030}, 0x24?)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:295 +0x108
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked(0x1401bcb7308, {0x10b8abdb0, 0x14004a82030}, 0x1400b60a5a0)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go:114 +0x1b4
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked(0x1401bcb7260, {0x10b8abdb0, 0x14004a82030}, 0x1400b60a5a0)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:246 +0x344
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send(0x1401bcb7088, {0x10b8abdb0, 0x14008771140}, 0x1400b60a5a0)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:535 +0x47c
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender(0x140019fa500, {0x10b8abdb0, 0x14008771140}, 0x1400b60a5a0, {0x13a8a75e0, 0x1401bcb7088})
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/db.go:1127 +0x98
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send(0x14003d481e0, {0x10b8abdb0, 0x14008771140}, 0x1400b60a5a0)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/txn.go:1282 +0x224
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv.sendAndFill({0x10b8abdb0, 0x14008771140}, 0x1400ff9a518, 0x14004350c08)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/db.go:959 +0xd8
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Run(0x14003d481e0, {0x10b8abdb0, 0x14008771140}, 0x14004350c08)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/txn.go:800 +0x6c
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.splitTxnAttempt({0x10b8abdb0, 0x14008771140}, 0x14001b5e008, 0x14003d481e0, 0x470, {0x14019b24e80, 0x1c, 0x1c}, {0x10b8ffbc0?, 0x1bcb7328?}, ...)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_command.go:208 +0x1f0
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).adminSplitWithDescriptor.func2({0x10b8abdb0?, 0x14008771140?}, 0x1048008d0?)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_command.go:444 +0x58
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec(0x14003d481e0, {0x10b8abdb0, 0x14008771140}, 0x1400ff9ae90)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/txn.go:1049 +0x74
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv.runTxn({0x10b8abdb0, 0x14008771140}, 0x14003d481e0, 0x1?)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/db.go:1092 +0x3c
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl(0x10b8abdb0?, {0x10b8abdb0, 0x14008771140}, 0x1?, 0x0?, 0x0, 0x1400ff9ae90)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/db.go:1055 +0xa8
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 !github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn(...)
F240320 10:02:12.236627 13807 kv/kvserver/replica_write.go:203 [T1,Vsystem,n1,s1,r1131/1:/Table/120/2/430001{526…-606…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/db.go:1030
```
Perhaps this will be helped by
https://github.com/cockroachdb/cockroach/pull/119894 or https://github.com/cockroachdb/cockroach/pull/119922 but even in the absence of those, it seems incorrect to translate a Request Cancelled error into a replica corruption error.
Jira issue: CRDB-36854
Contributor guide
Assessment
This issue has not been assessed yet.