cockroachdb / cockroachdb/cockroach

roachtest: ssh_problem failed

Open
#136,285 16 comments 0 reactions 0 assignees View on GitHub
branch-release-24.2.6-rc O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.ssh_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/17935206?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/17935206?buildTab=artifacts#/disk-stalled/wal-failover/among-stores) on release-24.2.6-rc @ [9dab6cb9f71f21ef1e9c98640193035783ba9a4e](https://github.com/cockroachdb/cockroach/commits/9dab6cb9f71f21ef1e9c98640193035783ba9a4e):

```
test disk-stalled/wal-failover/among-stores failed: (cluster.go:2398).Run: full command output in run_103035.501445936_n1_sudo-dmsetup-resume-.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/disk-stalled/wal-failover/among-stores/run_1/ssh/ssh_103035.599869792_n1_sudo-dmsetup-resume-.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cluster.go:2398).Run: context canceled
(cluster.go:2398).Run: context canceled
test artifacts and logs in: /artifacts/disk-stalled/wal-failover/among-stores/run_1
```

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=16
- encrypted=true
- fs=ext4
- localSSD=true
- runtimeAssertionsBuild=true
- ssd=2
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-17935206/disk-stalled-wal-failover-among-stores/1732700363886/1732703811000)

Same failure on other branches

- #136280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.17-rc]
- #136188 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc]
- #134939 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc]
- #134500 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc]
- #134374 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake]
- #134034 roachtest: ssh_problem failed [O-roachtest O-robot X-infra-flake]
- #133924 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake]
- #132980 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3]
- #131724 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc]
- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc]

/cc @cockroachdb/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*ssh_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-44955

Contributor guide

Open the contributing guide

Research direction

Start with the roachtest README and the TeamCity artifacts linked in the report, especially the full command output and retained SSH log under disk-stalled/wal-failover/among-stores. Compare the related ssh_problem failures listed across branches and the roachdash history; done means identifying and addressing the recurring failure pattern, with the affected roachtest no longer producing this flake.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.