cockroachdb / cockroachdb/cockroach
backup: TestOnlineRestoreStatementResult failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
backup.TestOnlineRestoreStatementResult [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_TestsAwsLinuxArm64_UnitTests/21438574?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_TestsAwsLinuxArm64_UnitTests/21438574?buildTab=artifacts#/) on master @ [35738542947a44831e63a7e11a3bff502dfa6ea2](https://github.com/cockroachdb/cockroach/commits/35738542947a44831e63a7e11a3bff502dfa6ea2):
```
| | pkg/util/stop/stopper.go:358
| | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
| | pkg/rpc/context.go:137
| | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func5
| | pkg/rpc/context.go:918
| | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
| | pkg/rpc/context.go:817
| | github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func13
| | pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:185
| | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.getChainUnaryInvoker.func4
| | pkg/rpc/context.go:1002
| | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3
| | pkg/rpc/context.go:887
| | github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch
| | pkg/rpc/context.go:1010
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch
| | pkg/kv/kvclient/kvcoord/transport.go:229
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext
| | pkg/kv/kvclient/kvcoord/transport.go:191
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas
| | pkg/kv/kvclient/kvcoord/dist_sender.go:2912
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch
| | pkg/kv/kvclient/kvcoord/dist_sender.go:2333
| Wraps: (2) scanning intents
| Wraps: (3) context canceled
| Error types: (1) *errbase.opaqueWrapper (2) *errutil.withPrefix (3) *errors.errorString
Wraps: (8) node unavailable; try another peer
Error types: (1) *withstack.withStack (2) *withstack.withStack (3) *errutil.withPrefix (4) *errbase.opaqueWrapper (5) *errutil.withPrefix (6) *kvpb.AmbiguousResultError (7) *secondary.withSecondaryError (8) *kvpb.NodeUnavailableError
goroutine 122785 [running]:
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0x4dfb68a28d40?, {0x6a6c258?, 0x4dfb70d27ea0?}, 0x4dfb6761f140?})
pkg/util/ctxgroup/ctxgroup.go:151 +0xc0
github.com/cockroachdb/cockroach/pkg/backup.(*restoreResumer).doDownloadFiles(0x4dfb695dd600, {0x6a6c290, 0x4dfb5bbf4120}, {0x6ad5bd0, 0x4dfb802908f0})
pkg/backup/restore_online.go:570 +0x518
github.com/cockroachdb/cockroach/pkg/backup.(*restoreResumer).doResume(0x4dfb695dd600, {0x6a6c290, 0x4dfb5bbf4120}, {0x611cbc0?, 0x4dfb802908f0})
pkg/backup/restore_job.go:2257 +0x240
github.com/cockroachdb/cockroach/pkg/backup.(*restoreResumer).Resume(0x4dfb695dd600?, {0x6a6c290?, 0x4dfb5bbf4120?}, {0x611cbc0?, 0x4dfb802908f0})
pkg/backup/restore_job.go:2237 +0xac
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2(0x4dfb693dd200?, 0x0?, 0x4dfb7163b8b0, {0x6a405b8?, 0x4dfb695dd600?}, {0x6a6c290?, 0x4dfb5bbf4120?}, {0x611cbc0?, 0x4dfb802908f0?})
pkg/jobs/registry.go:1690 +0xe8
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine(0x4dfb7e87c008, {0x6a6e658, 0x4dfb69fc9a40}, {0x611cbc0, 0x4dfb802908f0}, {0x6a405b8, 0x4dfb695dd600}, 0x4dfb693dd200, {0x4dfb6bd571f2, 0x7}, ...)
pkg/jobs/registry.go:1691 +0x838
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob(0x4dfb7e87c008, {0x6a6e658, 0x4dfb69fc9a40}, {0x6a405b8, 0x4dfb695dd600}, 0x4dfb693dd200, {0x4dfb6bd571f2, 0x7}, {0x4dfb6692f890, 0x17})
pkg/jobs/adopt.go:503 +0x3c8
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1({0x6a6e658?, 0x4dfb69fc9860?})
pkg/jobs/adopt.go:350 +0xe0
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0x6a6e658, 0x4dfb69fc9860})
pkg/util/stop/stopper.go:447 +0x7c
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 122896
pkg/util/stop/stopper.go:445 +0xe0
```
Help
See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)
/cc @cockroachlabs/disaster-recovery
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestOnlineRestoreStatementResult.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-66059
Contributor guide
Assessment
This issue has not been assessed yet.