cockroachdb / cockroachdb/cockroach

backup: TestBackupRestoreMultiNodeRemote failed

Open
#175,694 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-disaster-recovery branch-master C-test-failure O-robot release-blocker T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR metrics pending

Description

backup.TestBackupRestoreMultiNodeRemote failed on master @ 0ff013855fdee74d72243c42da24c01a62767848:

Data race:

WARNING: DATA RACE
Read at 0x00c03ebfd8f0 by goroutine 19500:
  github.com/cockroachdb/cockroach/pkg/util/hlc.(*Timestamp).Size()
      bazel-out/k8-fastbuild/bin/pkg/util/hlc/hlc_go_proto_/github.com/cockroachdb/cockroach/pkg/util/hlc/timestamp.pb.go:253 +0x145
  github.com/cockroachdb/cockroach/pkg/storage/enginepb.(*TxnMeta).Size()
      bazel-out/k8-fastbuild/bin/pkg/storage/enginepb/enginepb_go_proto_/github.com/cockroachdb/cockroach/pkg/storage/enginepb/mvcc3.pb.go:1969 +0xf7
  github.com/cockroachdb/cockroach/pkg/roachpb.(*Transaction).Size()
      bazel-out/k8-fastbuild/bin/pkg/roachpb/roachpb_go_proto_/github.com/cockroachdb/cockroach/pkg/roachpb/data.pb.go:4470 +0x4d
  github.com/cockroachdb/cockroach/pkg/kv/kvpb.(*Header).Size()
      bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:28237 +0x28a
  github.com/cockroachdb/cockroach/pkg/kv/kvpb.(*BatchRequest).Size()
      bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:28411 +0x37
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.readAll()
      pkg/kv/kvclient/kvcoord/transport_race.go:39 +0x34
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.startRacer.func1()
      pkg/kv/kvclient/kvcoord/transport_race.go:127 +0x157
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1()
      pkg/util/stop/stopper.go:447 +0x112
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.gowrap1()
      pkg/util/stop/stopper.go:448 +0x46

Previous write at 0x00c03ebfd8f0 by goroutine 2179:
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).canAttempt1PCEvaluation()
      pkg/kv/kvserver/replica_write.go:405 +0x69c
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatch()
      pkg/kv/kvserver/replica_write.go:448 +0x19b
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateProposal()
      pkg/kv/kvserver/replica_proposal.go:1096 +0x25a
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).requestToProposal()
      pkg/kv/kvserver/replica_proposal.go:1204 +0x14c
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evalAndPropose()
      pkg/kv/kvserver/replica_raft.go:129 +0x1da
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch()
      pkg/kv/kvserver/replica_write.go:190 +0xbb9
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries()
      pkg/kv/kvserver/replica_send.go:573 +0x682
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWorkStats()
      pkg/kv/kvserver/replica_send.go:224 +0xb44
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWorkStats()
      pkg/kv/kvserver/store_send.go:242 +0xc64
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWorkStats()
      pkg/kv/kvserver/stores.go:195 +0xfb
  github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal()
      pkg/server/node.go:1750 +0xbc9
  github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch()
      pkg/server/node.go:1943 +0x3f5
  github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1()
      pkg/rpc/context.go:822 +0x74
  github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func13()
      pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:60 +0x74b
  github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func5()
      pkg/rpc/context.go:933 +0x88
  github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3()
      pkg/rpc/context.go:197 +0xf0
  github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func5()
      pkg/rpc/context.go:933 +0x88
  github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor()
      pkg/rpc/auth.go:107 +0x335
  github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor-fm()
      <autogenerated>:1 +0xf0
  github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func5()
      pkg/rpc/context.go:933 +0x88
  github.com/cockroachdb/cockroach/pkg/rpc.gatewayRequestRecoveryInterceptor()
      pkg/rpc/metrics.go:826 +0x182
  github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func5()
      pkg/rpc/context.go:933 +0x88
  github.com/cockroachdb/cockroach/pkg/server.newGRPCServer.NewRequestMetricsInterceptor.func4()
      pkg/rpc/metrics.go:610 +0x3a1
  github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func5()
      pkg/rpc/context.go:933 +0x88
  github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1()
      pkg/rpc/context.go:154 +0x6a
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr()
      pkg/util/stop/stopper.go:358 +0x236
  github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1()
      pkg/rpc/context.go:152 +0x14b
  github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func5()
      pkg/rpc/context.go:933 +0x88
  github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2()
      pkg/rpc/context.go:832 +0x75
  github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func13()
      pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:185 +0x4fc
  github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.getChainUnaryInvoker.func4()
      pkg/rpc/context.go:1017 +0x18c
  github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3()
      pkg/rpc/context.go:902 +0x470
  github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch()
      pkg/rpc/context.go:1025 +0xd0
  github.com/cockroachdb/cockroach/pkg/rpc.(*internalClientAdapter).Batch()
      <autogenerated>:1 +0x1a
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch()
      pkg/kv/kvclient/kvcoord/transport.go:245 +0x458
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext()
      pkg/kv/kvclient/kvcoord/transport.go:194 +0x134
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext()
      pkg/kv/kvclient/kvcoord/transport_race.go:80 +0x181
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*raceTransport).SendNext()
      <autogenerated>:1 +0x5e
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas()
      pkg/kv/kvclient/kvcoord/dist_sender.go:2944 +0x2a5b
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch()
      pkg/kv/kvclient/kvcoord/dist_sender.go:2353 +0x924
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges()
      pkg/kv/kvclient/kvcoord/dist_sender.go:1899 +0xf92
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send()
      pkg/kv/kvclient/kvcoord/dist_sender.go:1403 +0x825
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked()
      pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:77 +0x258
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked()
      pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go:42 +0x13d
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts()
      pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:228 +0x82
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked()
      pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:162 +0x1bc
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked()
      pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go:190 +0x40e
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked()
      pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:340 +0x29c
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnWriteBuffer).SendLocked()
      pkg/kv/kvclient/kvcoord/txn_interceptor_write_buffer.go:298 +0x2cb
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked()
      pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go:124 +0x463
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked()
      pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:267 +0x7a2
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send()
      pkg/kv/kvclient/kvcoord/txn_coord_sender.go:574 +0xd74
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender()
      pkg/kv/db.go:1277 +0x17d
  github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send()
      pkg/kv/txn.go:1471 +0x6c5
  github.com/cockroachdb/cockroach/pkg/kv.(*Txn).commit()
      pkg/kv/txn.go:946 +0x1e6
  github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Commit()
      pkg/kv/txn.go:956 +0xa4
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).commitSQLTransactionInternal()
      pkg/sql/conn_executor_exec.go:2898 +0xba5
  github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).commitTxn()
      pkg/sql/internal.go:1497 +0x766
  github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).newInternalExecutorWithTxn.func1()
      pkg/sql/internal.go:1970 +0xeb
  github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn.func4()
      pkg/sql/internal.go:2122 +0x847
  github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec()
      pkg/kv/txn.go:1206 +0xad
  github.com/cockroachdb/cockroach/pkg/kv.runTxn()
      pkg/kv/db.go:1195 +0x6b
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl()
      pkg/kv/db.go:1154 +0x23e
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn()
      pkg/kv/db.go:1125 +0x4f
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn-fm()
      <autogenerated>:1 +0xe
  github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn()
      pkg/sql/internal.go:2092 +0x6d1
  github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).Txn()
      pkg/sql/internal.go:2010 +0xd4
  github.com/cockroachdb/cockroach/pkg/sql.(*internalDBWithOverrides).Txn()
      pkg/sql/internal.go:2177 +0x124
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).claimJobs()
      pkg/jobs/adopt.go:126 +0x328
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func5()
      pkg/jobs/registry.go:1061 +0x133
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
      pkg/jobs/registry.go:996 +0x322
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
      pkg/jobs/registry.go:986 +0x8a
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.(*Registry).Start.func1.func12()
      pkg/jobs/registry.go:1012 +0x4c
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
      pkg/jobs/registry.go:986 +0x8a
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.(*Registry).Start.func1.func12()
      pkg/jobs/registry.go:1012 +0x4c
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
      pkg/jobs/registry.go:996 +0x322
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
      pkg/jobs/registry.go:986 +0x8a
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.(*Registry).Start.func1.func12()
      pkg/jobs/registry.go:1012 +0x4c
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
      pkg/jobs/registry.go:986 +0x8a
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.(*Registry).Start.func1.func12()
      pkg/jobs/registry.go:1012 +0x4c
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +0x4e1
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1208 +0x4c6
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
      pkg/jobs/registry.go:1209 +

Jira issue: CRDB-68577

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing backup.TestBackupRestoreMultiNodeRemote with the race detector from the reported master commit. Trace the conflicting access through pkg/kv/kvclient/kvcoord/transport_race.go and pkg/kv/kvserver/replica_write.go, along with the reported generated Size calls; done means the test no longer reports this data race.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.