cockroachdb / cockroachdb/cockroach
roachtest: ssh_problem failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.
roachtest.ssh_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/17935206?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/17935206?buildTab=artifacts#/disk-stalled/wal-failover/among-stores) on release-24.2.6-rc @ [9dab6cb9f71f21ef1e9c98640193035783ba9a4e](https://github.com/cockroachdb/cockroach/commits/9dab6cb9f71f21ef1e9c98640193035783ba9a4e):
```
test disk-stalled/wal-failover/among-stores failed: (cluster.go:2398).Run: full command output in run_103035.501445936_n1_sudo-dmsetup-resume-.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/disk-stalled/wal-failover/among-stores/run_1/ssh/ssh_103035.599869792_n1_sudo-dmsetup-resume-.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cluster.go:2398).Run: context canceled
(cluster.go:2398).Run: context canceled
test artifacts and logs in: /artifacts/disk-stalled/wal-failover/among-stores/run_1
```
Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=16
- encrypted=true
- fs=ext4
- localSSD=true
- runtimeAssertionsBuild=true
- ssd=2
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-17935206/disk-stalled-wal-failover-among-stores/1732700363886/1732703811000)
Same failure on other branches
- #136280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.17-rc]
- #136188 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc]
- #134939 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc]
- #134500 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc]
- #134374 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake]
- #134034 roachtest: ssh_problem failed [O-roachtest O-robot X-infra-flake]
- #133924 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake]
- #132980 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3]
- #131724 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc]
- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc]
/cc @cockroachdb/test-eng
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*ssh_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-44955
Contributor guide
Research direction
Start with the roachtest README and the TeamCity artifacts linked in the report, especially the full command output and retained SSH log under disk-stalled/wal-failover/among-stores. Compare the related ssh_problem failures listed across branches and the roachdash history; done means identifying and addressing the recurring failure pattern, with the affected roachtest no longer producing this flake.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- infrastructure, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100