cockroachdb / cockroachdb/cockroach
roachtest: ssh_problem failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
roachtest.ssh_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/20377507?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/20377507?buildTab=artifacts#/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew) on release-24.3.19-rc @ [faf059d833b409c16409d1095dfa387068996c1c](https://github.com/cockroachdb/cockroach/commits/faf059d833b409c16409d1095dfa387068996c1c):
```
test allocbench/nodes=7/cpu=8/kv/r=95/ops=skew failed: (test_runner.go:829).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 ubuntu@34.74.85.190:./cockroach
Warning: Permanently added '34.74.85.190' (ECDSA) to the list of known hosts.
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]
```
Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=8
- encrypted=false
- fs=ext4
- localSSD=true
- runtimeAssertionsBuild=false
- ssd=0
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-20377507/allocbench-nodes-7-cpu-8-kv-r-95-ops-skew/-62135596800000/-62135596680000)
Same failure on other branches
- #152156 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.3.1-rc]
- #151647 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.5-rc]
- #151588 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.23-rc]
- #150652 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.3.0-rc]
- #150125 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.1.9-rc]
- #150096 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.21-rc]
- #149972 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.16-rc]
- #149737 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.3-rc]
- #149125 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.3]
- #148346 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.15-rc]
/cc @cockroachdb/test-eng
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*ssh_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-53772
Contributor guide
Research direction
Start with the failure at test_runner.go:829 and the cluster.PutE scp path, then read the roachtest README and compare the repeated ssh_problem issues listed in this report. Use the linked TeamCity logs, artifacts, and Grafana view to determine the cause of the broken pipe. Done means identifying and addressing the recurring failure so the roachtest no longer reproduces it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- infrastructure, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100