backup: TestBackupRestoreExecLocality failed

backup.TestBackupRestoreExecLocality [failed](https://mesolite.cluster.engflow.com/invocations/default/cad1800d-6c06-40c0-accb-ff3cb057ca10?testReportRun=3&testReportShard=31&testReportAttempt=1#targets-Ly9wa2cvYmFja3VwOmJhY2t1cF90ZXN0) on master @ [5109832ee779a0bf9d147f54e164085d92ea6d4c](https://github.com/cockroachdb/cockroach/commits/5109832ee779a0bf9d147f54e164085d92ea6d4c):

```
github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0x113dd5094000, {0x69fb420?, 0xdc6ed00?})
external/com_github_cockroachdb_pebble/record/log_writer.go:677 +0x195
runtime/pprof.Do({0x7b953c0?, 0xdc6ed00?}, {{0x113da215e0a0?, 0x113da378cfd0?, 0x499db0?}}, 0x113da378cfc0)
GOROOT/src/runtime/pprof/runtime.go:57 +0x8c
github.com/cockroachdb/pebble/record.NewLogWriter.func2()
external/com_github_cockroachdb_pebble/record/log_writer.go:591 +0x50
created by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 119690968
external/com_github_cockroachdb_pebble/record/log_writer.go:590 +0x4c5

goroutine 3909514 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0x113da2cd43a0, 0x44af)
GOROOT/src/runtime/sema.go:617 +0x1b3
sync.(*Cond).Wait(0x113dbe911df8?)
GOROOT/src/sync/cond.go:71 +0x73
github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/sidetransport.(*updatesBuf).GetBySeq(0x113da2cd4340, {0x7b95c98, 0x113dd6e5cc80}, 0x2dfc)
pkg/kv/kvserver/closedts/sidetransport/sender.go:759 +0x117
github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/sidetransport.(*rpcConn).run.func1({0x7b98070?, 0x113da1f123c0?})
pkg/kv/kvserver/closedts/sidetransport/sender.go:983 +0x285
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0x7b98070, 0x113da1f123c0})
pkg/util/stop/stopper.go:447 +0x8a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 563262
pkg/util/stop/stopper.go:445 +0x125

goroutine 3909539 [select, 39 minutes]:
storj.io/drpc/drpcmanager.(*Manager).manageStream(0x499db0?, {0x7b95cd0, 0x113dc8f09ad0}, 0x113dd119c708)
external/io_storj_drpc/drpcmanager/manager.go:304 +0x13d
created by storj.io/drpc/drpcmanager.(*Manager).newStream in goroutine 511595
external/io_storj_drpc/drpcmanager/manager.go:288 +0x307

goroutine 119588961 [select]:
github.com/cockroachdb/pebble.(*DB).maybeScheduleDelayedFlush.func1()
external/com_github_cockroachdb_pebble/compaction.go:1480 +0xfd
created by github.com/cockroachdb/pebble.(*DB).maybeScheduleDelayedFlush in goroutine 378774
external/com_github_cockroachdb_pebble/compaction.go:1476 +0x170

goroutine 119587789 [sync.Cond.Wait, "pebble":"wal-sync"]:
sync.runtime_notifyListWait(0x113dbeea8088, 0x55)
GOROOT/src/runtime/sema.go:617 +0x1b3
sync.(*Cond).Wait(0x113dbeea8000?)
GOROOT/src/sync/cond.go:71 +0x73
github.com/cockroachdb/pebble/record.(*flusherCond).Wait(...)
external/com_github_cockroachdb_pebble/record/log_writer.go:374
github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0x113dbeea8000, {0x69fb420?, 0xdc6ed00?})
external/com_github_cockroachdb_pebble/record/log_writer.go:677 +0x195
runtime/pprof.Do({0x7b953c0?, 0xdc6ed00?}, {{0x113da215e0a0?, 0x113da1f4afd0?, 0x499db0?}}, 0x113da1f4afc0)
GOROOT/src/runtime/pprof/runtime.go:57 +0x8c
github.com/cockroachdb/pebble/record.NewLogWriter.func2()
external/com_github_cockroachdb_pebble/record/log_writer.go:591 +0x50
created by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 119587680
external/com_github_cockroachdb_pebble/record/log_writer.go:590 +0x4c5
```

Parameters:
- attempt=1
- deadlock=true
- run=3
- shard=31
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachlabs/disaster-recovery

[Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-67882