cockroachdb / cockroachdb/cockroach

roachtest: ssh_problem failed

Open
#152,321 6 comments 0 reactions 0 assignees View on GitHub
branch-release-24.3.19-rc O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.ssh_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/20377507?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/20377507?buildTab=artifacts#/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew) on release-24.3.19-rc @ [faf059d833b409c16409d1095dfa387068996c1c](https://github.com/cockroachdb/cockroach/commits/faf059d833b409c16409d1095dfa387068996c1c):

```
test allocbench/nodes=7/cpu=8/kv/r=95/ops=skew failed: (test_runner.go:829).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 ubuntu@34.74.85.190:./cockroach
Warning: Permanently added '34.74.85.190' (ECDSA) to the list of known hosts.
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]
```

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=8
- encrypted=false
- fs=ext4
- localSSD=true
- runtimeAssertionsBuild=false
- ssd=0
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-20377507/allocbench-nodes-7-cpu-8-kv-r-95-ops-skew/-62135596800000/-62135596680000)

Same failure on other branches

- #152156 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.3.1-rc]
- #151647 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.5-rc]
- #151588 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.23-rc]
- #150652 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.3.0-rc]
- #150125 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.1.9-rc]
- #150096 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.21-rc]
- #149972 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.16-rc]
- #149737 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.3-rc]
- #149125 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.3]
- #148346 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.15-rc]

/cc @cockroachdb/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*ssh_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-53772

Contributor guide

Open the contributing guide

Research direction

Start with the failure at test_runner.go:829 and the cluster.PutE scp path, then read the roachtest README and compare the repeated ssh_problem issues listed in this report. Use the linked TeamCity logs, artifacts, and Grafana view to determine the cause of the broken pipe. Done means identifying and addressing the recurring failure so the roachtest no longer reproduces it.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.