cockroachdb / cockroachdb/cockroach

roachtest: restore/online/tpcc-5k/azure/nodes=4/cpus=8 failed

Open
#175,506 1 comment 0 reactions 0 assignees View on GitHub
A-disaster-recovery branch-master C-test-failure O-roachtest O-robot release-blocker T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.restore/online/tpcc-5k/azure/nodes=4/cpus=8 [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAzureBazel/21516771?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAzureBazel/21516771?buildTab=artifacts#/restore/online/tpcc-5k/azure/nodes=4/cpus=8) on master @ [b112dc518c826da8b4b577934d4ab5c0c24da045](https://github.com/cockroachdb/cockroach/commits/b112dc518c826da8b4b577934d4ab5c0c24da045):

Failed with:

```
(assertions.go:388).Fail:
Error Trace: pkg/cmd/roachtest/tests/online_restore.go:652
pkg/cmd/roachtest/tests/online_restore.go:277
pkg/cmd/roachtest/test_runner.go:1992
src/runtime/asm_amd64.s:1771
Error: Received unexpected error:
link phase took too long: 2m34.405328438s greater than timeout 2m0s
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.executeORCmd
| pkg/cmd/roachtest/tests/online_restore.go:709
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runOnlineRestore
| pkg/cmd/roachtest/tests/online_restore.go:651
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerOnlineRestorePerf.func1
| pkg/cmd/roachtest/tests/online_restore.go:277
| main.(*testRunner).runTest.func2
| pkg/cmd/roachtest/test_runner.go:1992
| runtime.goexit
| src/runtime/asm_amd64.s:1771
Wraps: (2) link phase took too long: 2m34.405328438s greater than timeout 2m0s
Error types: (1) *withstack.withStack (2) *errutil.leafError
Test: restore/online/tpcc-5k/azure/nodes=4/cpus=8
Messages: failed to execute restore phase
(require.go:1376).NoError: FailNow called
test artifacts and logs in: /artifacts/restore/online/tpcc-5k/azure/nodes=4/cpus=8/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21516771-1789451590-110-n5cpu8-0001 | 40.76.225.188 | 10.1.0.160 |
| teamcity-21516771-1789451590-110-n5cpu8-0002 | 20.106.192.148 | 10.1.0.161 |
| teamcity-21516771-1789451590-110-n5cpu8-0003 | 74.235.161.230 | 10.1.0.163 |
| teamcity-21516771-1789451590-110-n5cpu8-0004 | 138.91.105.204 | 10.1.0.162 |
| teamcity-21516771-1789451590-110-n5cpu8-0005 | 172.210.10.180 | 10.1.0.114 |

Parameters:
- arch=amd64
- cloud=azure
- coverageBuild=false
- cpu=8
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=false
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21516771/restore-online-tpcc-5k-azure-nodes-4-cpus-8/1789464955422/1789465767050)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21516771-1789451590-110-n5cpu8&from_ts=1789464955422&to_ts=1789465767050&live=false&storage=flex_tier)

/cc @cockroachlabs/disaster-recovery

[Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-68422

Contributor guide

Open the contributing guide

Research direction

Start with pkg/cmd/roachtest/tests/online_restore.go at lines 277, 652, and 709, then inspect the linked TeamCity artifacts and logs for the link-phase timeout. Use the roachtest README and the Grafana or Datadog links to determine why the phase exceeded two minutes. Done means the restore/online/tpcc-5k/azure/nodes=4/cpus=8 test completes without this timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, go
Domain
cloud, databases, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.