cockroachdb / cockroachdb/cockroach

backup: TestBackupRestorePartitionedMergeDirectories failed

Open
#175,561 1 comment 0 reactions 0 assignees View on GitHub
A-disaster-recovery branch-master C-test-failure O-robot release-blocker T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

backup.TestBackupRestorePartitionedMergeDirectories [failed](https://mesolite.cluster.engflow.com/invocations/default/133acc16-45c4-405c-961c-646a6c2b2f80?testReportRun=2&testReportShard=35&testReportAttempt=1#targets-Ly9wa2cvYmFja3VwOmJhY2t1cF90ZXN0) on master @ [0d39086a7c6c0cca4e832119a3c828463eb2534e](https://github.com/cockroachdb/cockroach/commits/0d39086a7c6c0cca4e832119a3c828463eb2534e):

Data race:

```
WARNING: DATA RACE
Write at 0x00c04dfc4b70 by goroutine 2101:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).canAttempt1PCEvaluation()
pkg/kv/kvserver/replica_write.go:405 +0x69c
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatch()
pkg/kv/kvserver/replica_write.go:448 +0x19b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateProposal()
pkg/kv/kvserver/replica_proposal.go:1096 +0x25a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).requestToProposal()
pkg/kv/kvserver/replica_proposal.go:1204 +0x14c
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evalAndPropose()
pkg/kv/kvserver/replica_raft.go:129 +0x1da
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch()
pkg/kv/kvserver/replica_write.go:190 +0xbb9
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries()
pkg/kv/kvserver/replica_send.go:573 +0x682
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWorkStats()
pkg/kv/kvserver/replica_send.go:224 +0xb44
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWorkStats()
pkg/kv/kvserver/store_send.go:242 +0xc64
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWorkStats()
pkg/kv/kvserver/stores.go:195 +0xfb
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal()
pkg/server/node.go:1750 +0xbc9
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch()
pkg/server/node.go:1943 +0x3f5
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1()
pkg/rpc/context.go:822 +0x74
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func13()
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:60 +0x74b
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func5()
pkg/rpc/context.go:933 +0x88
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3()
pkg/rpc/context.go:197 +0xf0
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func5()
pkg/rpc/context.go:933 +0x88
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor()
pkg/rpc/auth.go:107 +0x335
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor-fm()
:1 +0xf0
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func5()
pkg/rpc/context.go:933 +0x88
github.com/cockroachdb/cockroach/pkg/rpc.gatewayRequestRecoveryInterceptor()
pkg/rpc/metrics.go:816 +0x182
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func5()
pkg/rpc/context.go:933 +0x88
github.com/cockroachdb/cockroach/pkg/server.newGRPCServer.NewRequestMetricsInterceptor.func4()
pkg/rpc/metrics.go:600 +0x3a1
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func5()
pkg/rpc/context.go:933 +0x88
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1()
pkg/rpc/context.go:154 +0x6a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr()
pkg/util/stop/stopper.go:358 +0x236
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1()
pkg/rpc/context.go:152 +0x14b
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func5()
pkg/rpc/context.go:933 +0x88
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2()
pkg/rpc/context.go:832 +0x75
github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func13()
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:185 +0x4fc
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.getChainUnaryInvoker.func4()
pkg/rpc/context.go:1017 +0x18c
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3()
pkg/rpc/context.go:902 +0x470
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch()
pkg/rpc/context.go:1025 +0xd0
github.com/cockroachdb/cockroach/pkg/rpc.(*internalClientAdapter).Batch()
:1 +0x1a
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch()
pkg/kv/kvclient/kvcoord/transport.go:245 +0x458
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext()
pkg/kv/kvclient/kvcoord/transport.go:194 +0x134
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext()
pkg/kv/kvclient/kvcoord/transport_race.go:80 +0x181
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*raceTransport).SendNext()
:1 +0x5e
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas()
pkg/kv/kvclient/kvcoord/dist_sender.go:2941 +0x2a5b
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch()
pkg/kv/kvclient/kvcoord/dist_sender.go:2350 +0x924
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges()
pkg/kv/kvclient/kvcoord/dist_sender.go:1896 +0xf92
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send()
pkg/kv/kvclient/kvcoord/dist_sender.go:1400 +0x825
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked()
pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:77 +0x258
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked()
pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go:42 +0x13d
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts()
pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:228 +0x82
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked()
pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:162 +0x1bc
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked()
pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go:190 +0x40e
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked()
pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:340 +0x29c
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnWriteBuffer).SendLocked()
pkg/kv/kvclient/kvcoord/txn_interceptor_write_buffer.go:298 +0x2cb
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked()
pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go:124 +0x463
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked()
pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:267 +0x7a2
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send()
pkg/kv/kvclient/kvcoord/txn_coord_sender.go:574 +0xd74
github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender()
pkg/kv/db.go:1277 +0x17d
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send()
pkg/kv/txn.go:1463 +0x6c5
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).commit()
pkg/kv/txn.go:942 +0x1e6
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Commit()
pkg/kv/txn.go:952 +0xa4
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).commitSQLTransactionInternal()
pkg/sql/conn_executor_exec.go:2899 +0xb85
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).commitTxn()
pkg/sql/internal.go:1496 +0x766
github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).newInternalExecutorWithTxn.func1()
pkg/sql/internal.go:1959 +0xeb
github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn.func4()
pkg/sql/internal.go:2111 +0x847
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec()
pkg/kv/txn.go:1202 +0xad
github.com/cockroachdb/cockroach/pkg/kv.runTxn()
pkg/kv/db.go:1195 +0x6b
github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl()
pkg/kv/db.go:1154 +0x23e
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn()
pkg/kv/db.go:1125 +0x4f
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn-fm()
:1 +0xe
github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn()
pkg/sql/internal.go:2081 +0x6d1
github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).Txn()
pkg/sql/internal.go:1999 +0xd4
github.com/cockroachdb/cockroach/pkg/sql.(*internalDBWithOverrides).Txn()
pkg/sql/internal.go:2166 +0x124
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).claimJobs()
pkg/jobs/adopt.go:126 +0x328
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func5()
pkg/jobs/registry.go:1061 +0x133
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
pkg/jobs/registry.go:996 +0x322
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
pkg/jobs/registry.go:986 +0x8a
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.(*Registry).Start.func1.func12()
pkg/jobs/registry.go:1012 +0x4c
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
pkg/jobs/registry.go:986 +0x8a
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.(*Registry).Start.func1.func12()
pkg/jobs/registry.go:1012 +0x4c
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
pkg/jobs/registry.go:986 +0x8a
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.(*Registry).Start.func1.func12()
pkg/jobs/registry.go:1012 +0x4c
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
pkg/jobs/registry.go:986 +0x8a
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.(*Registry).Start.func1.func12()
pkg/jobs/registry.go:1012 +0x4c
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
pkg/jobs/registry.go:986 +0x8a
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.(*Registry).Start.func1.func12()
pkg/jobs/registry.go:1012 +0x4c
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
pkg/jobs/registry.go:986 +0x8a
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.(*Registry).Start.func1.func12()
pkg/jobs/registry.go:1012 +0x4c
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
pkg/jobs/registry.go:986 +0x8a
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.(*Registry).Start.func1.func12()
pkg/jobs/registry.go:1012 +0x4c
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
pkg/jobs/registry.go:986 +0x8a
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.(*Registry).Start.func1.func12()
pkg/jobs/registry.go:1012 +0x4c
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
pkg/jobs/registry.go:986 +0x8a
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.(*Registry).Start.func1.func12()
pkg/jobs/registry.go:1012 +0x4c
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
pkg/jobs/registry.go:986 +0x8a
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.(*Registry).Start.func1.func12()
pkg/jobs/registry.go:1012 +0x4c
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
pkg/jobs/registry.go:986 +0x8a
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.(*Registry).Start.func1.func12()
pkg/jobs/registry.go:1012 +0x4c
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
pkg/jobs/registry.go:986 +0x8a
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.(*Registry).Start.func1.func12()
pkg/jobs/registry.go:1012 +0x4c
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
pkg/jobs/registry.go:986 +0x8a
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.(*Registry).Start.func1.func12()
pkg/jobs/registry.go:1012 +0x4c
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
pkg/jobs/registry.go:996 +0x322
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
pkg/jobs/registry.go:986 +0x8a
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.(*Registry).Start.func1.func12()
pkg/jobs/registry.go:1012 +0x4c
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
pkg/jobs/registry.go:986 +0x8a
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.(*Registry).Start.func1.func12()
pkg/jobs/registry.go:1012 +0x4c
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
pkg/jobs/registry.go:986 +0x8a
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.(*Registry).Start.func1.func12()
pkg/jobs/registry.go:1012 +0x4c
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
pkg/jobs/registry.go:996 +0x322
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
pkg/jobs/registry.go:986 +0x8a
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.(*Registry).Start.func1.func12()
pkg/jobs/registry.go:1012 +0x4c
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
pkg/jobs/registry.go:996 +0x322
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
pkg/jobs/registry.go:986 +0x8a
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.(*Registry).Start.func1.func12()
pkg/jobs/registry.go:1012 +0x4c
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1208 +0x4c6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
pkg/jobs/registry.go:1209 +0x4e1

Previous read at 0x00c04dfc4b70 by goroutine 37453:
github.com/cockroachdb/cockroach/pkg/util/hlc.(*Timestamp).Size()
bazel-out/k8-fastbuild/bin/pkg/util/hlc/hlc_go_proto_/github.com/cockroachdb/cockroach/pkg/util/hlc/timestamp.pb.go:253 +0x145
github.com/cockroachdb/cockroach/pkg/storage/enginepb.(*TxnMeta).Size()
bazel-out/k8-fastbuild/bin/pkg/storage/enginepb/enginepb_go_proto_/github.com/cockroachdb/cockroach/pkg/storage/enginepb/mvcc3.pb.go:1969 +0xf7
github.com/cockroachdb/cockroach/pkg/roachpb.(*Transaction).Size()
bazel-out/k8-fastbuild/bin/pkg/roachpb/roachpb_go_proto_/github.com/cockroachdb/cockroach/pkg/roachpb/data.pb.go:4470 +0x4d
github.com/cockroachdb/cockroach/pkg/kv/kvpb.(*Header).Size()
bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:28237 +0x28a
github.com/cockroachdb/cockroach/pkg/kv/kvpb.(*BatchRequest).Size()
bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:28411 +0x37
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.readAll()
pkg/kv/kvclient/kvcoord/transport_race.go:39 +0x34
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.startRacer.func1()
pkg/kv/kvclient/kvcoord/transport_race.go:127 +0x157
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1()
pkg/util/stop/stopper.go:447 +0x112
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.gowrap1()
pkg/util/stop/stopper.go:448 +0x46

Goroutine 2101 (running) created at:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx()
pkg/util/stop/stopper.go:445 +0x246
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
pkg/util/stop/stopper.go:368 +0xce9
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start()
pkg/jobs/registry.go:1178 +0xae5
github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart()
pkg/server/server_sql.go:1744 +0x14a4
github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart()
pkg/server/server_sql.go:1656 +0xbed
github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart()
pkg/server/server.go:2366 +0x6bca
github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart()
pkg/server/server.go:2196 +0x4dd9
github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart()
pkg/server/server.go:1862 +0x248f
github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart()
pkg/server/testserver.go:959 +0x144
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).PreStart()
bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:19 +0x71
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start()
pkg/testutils/testcluster/testcluster.go:575 +0x627
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.StartTestCluster()
pkg/testutils/testcluster/testcluster.go:359 +0x91
github.com/cockroachdb/cockroach/pkg/backup/backuptestutils.StartBackupRestoreTestCluster()
pkg/backup/backuptestutils/testutils.go:201 +0x37c
github.com/cockroachdb/cockroach/pkg/backup.backupRestoreTestSetupWithParams()
pkg/backup/utils_test.go:75 +0x1be
github.com/cockroachdb/cockroach/pkg/backup.backupRestoreTestSetup()
pkg/backup/utils_test.go:84 +0x8b
github.com/cockroachdb/cockroach/pkg/backup.TestBackupRestorePartitionedMergeDirectories()
pkg/backup/backup_test.go:827 +0x13b
testing.tRunner()
GOROOT/src/testing/testing.go:2036 +0x21c
testing.(*T).Run.gowrap1()
GOROOT/src/testing/testing.go:2101 +0x38

Goroutine 37453 (running) created at:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicateQueue).processOneChangeWithTracing()
pkg/kv/kvserver/replicate_queue.go:860 +0x224
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicateQueue).process()
pkg/kv/kvserver/replicate_queue.go:738 +0x9b6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica.func1()
pkg/kv/kvserver/queue.go:1254 +0x127
github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout()
pkg/util/timeutil/timeout.go:28 +0x11e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica()
pkg/kv/kvserver/queue.go:1247 +0x6dc
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).DrainQueue()
pkg/kv/kvserver/queue.go:1738 +0x273
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicateQueue).processOneChangeWithTracing()
pkg/kv/kvserver/replicate_queue.go:860 +0x224
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicateQueue).process()
pkg/kv/kvserver/replicate_queue.go:738 +0x9b6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica.func1()
pkg/kv/kvserver/queue.go:1254 +0x127
github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout()
pkg/util/timeutil/timeout.go:28 +0x11e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica()
pkg/kv/kvserver/queue.go:1247 +0x6dc
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).DrainQueue()
pkg/kv/kvserver/queue.go:1738 +0x273
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicateQueue).processOneChangeWithTracing()
pkg/kv/kvserver/replicate_queue.go:860 +0x224
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicateQueue).process()
pkg/kv/kvserver/replicate_queue.go:738 +0x9b6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica.func1()
pkg/kv/kvserver/queue.go:1254 +0x127
github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout()
pkg/util/timeutil/timeout.go:28 +0x11e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica()
pkg/kv/kvserver/queue.go:1247 +0x6dc
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).DrainQueue()
pkg/kv/kvserver/queue.go:1738 +0x273
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicateQueue).processOneChangeWithTracing()
pkg/kv/kvserver/replicate_queue.go:860 +0x224
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicateQueue).process()
pkg/kv/kvserver/replicate_queue.go:738 +0x9b6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica.func1()
pkg/kv/kvserver/queue.go:1254 +0x127
github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout()
pkg/util/timeutil/timeout.go:28 +0x11e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica()
pkg/kv/kvserver/queue.go:1247 +0x6dc
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).DrainQueue()
pkg/kv/kvserver/queue.go:1738 +0x273
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicateQueue).processOneChangeWithTracing()
pkg/kv/kvserver/replicate_queue.go:860 +0x224
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicateQueue).process()
pkg/kv/kvserver/replicate_queue.go:738 +0x9b6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica.func1()
pkg/kv/kvserver/queue.go:1254 +0x127
github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout()
pkg/util/timeutil/timeout.go:28 +0x11e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica()
pkg/kv/kvserver/queue.go:1247 +0x6dc
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).DrainQueue()
pkg/kv/kvserver/queue.go:1738 +0x273
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicateQueue).processOneChangeWithTracing()
pkg/kv/kvserver/replicate_queue.go:860 +0x224
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicateQueue).process()
pkg/kv/kvserver/replicate_queue.go:738 +0x9b6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica.func1()
pkg/kv/kvserver/queue.go:1254 +0x127
github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout()
pkg/util/timeutil/timeout.go:28 +0x11e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica()
pkg/kv/kvserver/queue.go:1247 +0x6dc
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).DrainQueue()
pkg/kv/kvserver/queue.go:1738 +0x273
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicateQueue).processOneChangeWithTracing()
pkg/kv/kvserver/replicate_queue.go:860 +0x224
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicateQueue).process()
pkg/kv/kvserver/replicate_queue.go:738 +0x9b6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica.func1()
pkg/kv/kvserver/queue.go:1254 +0x127
github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout()
pkg/util/timeutil/timeout.go:28 +0x11e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica()
pkg/kv/kvserver/queue.go:1247 +0x6dc
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).DrainQueue()
pkg/kv/kvserver/queue.go:1738 +0x273
github.com/cockroachdb/cockroach/pkg/kv/kvserver.forceScanAndProcess()
pkg/kv/kvserver/queue_helpers_testutil.go:47 +0x179
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).ForceReplicationScanAndProcess()
pkg/kv/kvserver/queue_helpers_testutil.go:60 +0x124
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).WaitForFullReplication.func2()
pkg/testutils/testcluster/testcluster.go:1939 +0x125
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).VisitStores.func1()
pkg/kv/kvserver/stores.go:149 +0x41
github.com/cockroachdb/cockroach/pkg/util/syncutil.(*Map[go.shape.int32,go.shape.0f39f97b489fd25e35cddf431996da5daf625673f3c462eddb79fcf8e06911cd]).Range()
pkg/util/syncutil/map.go:391 +0x203
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).VisitStores()
pkg/kv/kvserver/stores.go:148 +0x94
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).WaitForFullReplication()
pkg/testutils/testcluster/testcluster.go:1931 +0x5d2
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start()
pkg/testutils/testcluster/testcluster.go:616 +0xaf2
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/server/status.eachRecordableValue()
recorder.go:1086 +0x1192
github.com/cockroachdb/cockroach/pkg/server/status.(*MetricsRecorder).GenerateNodeStatus()
recorder.go:941 +0x1096
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/server/status.eachRecordableValue()
recorder.go:1086 +0xd45
github.com/cockroachdb/cockroach/pkg/server/status.(*MetricsRecorder).GenerateNodeStatus()
recorder.go:928 +0xc26
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each()
pkg/util/metric/registry.go:265 +0xf7
githu

Jira issue: CRDB-68459

Contributor guide

Open the contributing guide

Research direction

Start with backup.TestBackupRestorePartitionedMergeDirectories and reproduce it with the race detector. Inspect pkg/kv/kvserver/replica_write.go, especially canAttempt1PCEvaluation, alongside the reported call paths in replica_proposal.go and replica_raft.go. Done means the test passes without reporting the data race.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.