cockroachdb / cockroachdb/cockroach

roachtest: ssh_problem failed

Open
#153,147 11 comments 0 reactions 0 assignees View on GitHub
branch-release-24.3.20-rc O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.ssh_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/20440840?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/20440840?buildTab=artifacts#/ledger/nodes=6/multi-az) on release-24.3.20-rc @ [ddd5aa35d2deefa82d9d50c5d5b3a6c53e208ad1](https://github.com/cockroachdb/cockroach/commits/ddd5aa35d2deefa82d9d50c5d5b3a6c53e208ad1):

```
test ledger/nodes=6/multi-az failed: (test_runner.go:829).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 ubuntu@34.16.73.251:./cockroach
Warning: Permanently added '34.16.73.251' (ECDSA) to the list of known hosts.
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]
```

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=16
- encrypted=false
- fs=ext4
- localSSD=true
- runtimeAssertionsBuild=false
- ssd=0
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-20440840/ledger-nodes-6-multi-az/-62135596800000/-62135596680000)

Same failure on other branches

- #153140 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.6-rc]
- #152951 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.3.2-rc]
- #152321 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.19-rc]
- #152156 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.3.1-rc]
- #151647 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.5-rc]
- #151588 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.23-rc]
- #150652 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.3.0-rc]
- #150125 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.1.9-rc]
- #150096 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.21-rc]
- #149972 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.16-rc]

/cc @cockroachdb/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*ssh_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-54236

Contributor guide

Open the contributing guide

Research direction

Start with the linked TeamCity log and artifacts for the roachtest.ssh_problem failure, then read pkg/cmd/roachtest/README.md and the linked investigation guide. Compare the repeated failures listed across branches and use the Grafana link for the affected run. Done requires identifying and addressing the recurring SSH transfer failure, with the relevant roachtest passing.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
infrastructure, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.