cockroachdb / cockroachdb/cockroach

roachtest: ssh_problem failed

Open
#136,346 20 comments 0 reactions 0 assignees View on GitHub
branch-release-24.3.1-rc O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.ssh_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAwsBazel/17949991?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAwsBazel/17949991?buildTab=artifacts#/ycsb/B/nodes=3/cpu=32/mvcc-range-keys=global) on release-24.3.1-rc @ [1d0b043f23ed67fa7ae6bfc220008feceb300aa4](https://github.com/cockroachdb/cockroach/commits/1d0b043f23ed67fa7ae6bfc220008feceb300aa4):

```
test ycsb/B/nodes=3/cpu=32/mvcc-range-keys=global failed: (cluster.go:2455).Run: full command output in run_081041.934093923_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/ycsb/B/nodes=3/cpu=32/mvcc-range-keys=global/run_1/ssh/ssh_081041.973007171_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/B/nodes=3/cpu=32/mvcc-range-keys=global/run_1
```

Parameters:
- arch=amd64
- cloud=aws
- coverageBuild=false
- cpu=32
- encrypted=false
- fs=ext4
- localSSD=true
- runtimeAssertionsBuild=false
- ssd=0
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

_Grafana is not yet available for aws clusters_

Same failure on other branches

- #136285 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.6-rc]
- #136280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.17-rc]
- #136188 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc]
- #134939 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc]
- #134500 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc]
- #134374 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake]
- #134034 roachtest: ssh_problem failed [O-roachtest O-robot X-infra-flake]
- #133924 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake]
- #132980 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3]
- #131724 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc]

/cc @cockroachdb/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*ssh_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-45032

Contributor guide

Open the contributing guide

Research direction

Start with the roachtest README, the linked TeamCity build, and the retained SSH and command logs under the listed artifacts path. Compare this report with the repeated ssh_problem issues linked in the body; the report does not define a specific change or acceptance criteria, so completion needs a maintainer-defined diagnosis or fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
infrastructure, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.