cockroachdb / cockroachdb/cockroach
roachtest: ssh_problem failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.
roachtest.ssh_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/20164999?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/20164999?buildTab=artifacts#/follower-reads/survival=zone/locality=global/reads=exact-staleness) on release-25.1.9-rc @ [da9b0ac6f6220c719b417a4e15d975773a8e74be](https://github.com/cockroachdb/cockroach/commits/da9b0ac6f6220c719b417a4e15d975773a8e74be):
```
test follower-reads/survival=zone/locality=global/reads=exact-staleness failed: (test_runner.go:823).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach-ea.linux-amd64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach-ea.linux-amd64 ubuntu@34.39.21.102:./cockroach
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]
```
Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- encrypted=false
- runtimeAssertionsBuild=true
- ssd=0
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-20164999/follower-reads-survival-zone-locality-global-reads-exact-stalen/-62135596800000/-62135596680000)
Same failure on other branches
- #150096 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.21-rc]
- #149972 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.16-rc]
- #149737 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.3-rc]
- #149125 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.3]
- #148346 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.15-rc]
- #148333 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.2-rc]
- #148257 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.20-rc]
- #148191 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.27-rc]
- #148064 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.1.8-rc]
- #147561 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]
/cc @cockroachdb/test-eng
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*ssh_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-52576
Contributor guide
Research direction
Start with pkg/cmd/roachtest/README.md and the failure at test_runner.go:823. Review the linked TeamCity build, artifacts, and Grafana data to trace the ssh_problem during cluster.PutE. Done means identifying the cause and verifying that the affected roachtest no longer fails with this SSH error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, go
- Domain
- cloud, infrastructure, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100