cockroachdb / cockroachdb/cockroach

roachtest: ssh_problem failed

Open
#163,913 0 comments 0 reactions 0 assignees View on GitHub
branch-release-25.2.14-rc O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.ssh_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21127061?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21127061?buildTab=artifacts#/backup/2TB/n10cpu4) on release-25.2.14-rc @ [41623f2c082b258db3f86bf01f6371ce96245f97](https://github.com/cockroachdb/cockroach/commits/41623f2c082b258db3f86bf01f6371ce96245f97):

```
test backup/2TB/n10cpu4 failed: (test_runner.go:840).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 ubuntu@136.119.145.10:./cockroach
Warning: Permanently added '136.119.145.10' (ECDSA) to the list of known hosts.
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]
```

Parameters:
- arch=arm64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=false
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21127061/backup-2tb-n10cpu4/-62135596800000/-62135596680000)

Same failure on other branches

- #162299 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.26-rc]
- #161667 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.3.8-rc]
- #161664 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.12-rc]
- #161602 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.4.4-rc]
- #161498 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.1.0-rc]
- #160756 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.3.7-rc]
- #160426 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.4.3-rc]
- #160363 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.11-rc]
- #160157 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.25-rc]
- #159400 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.1]

/cc @cockroachdb/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*ssh_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-60520

Contributor guide

Open the contributing guide

Research direction

Start with pkg/cmd/roachtest/README.md and the failure at test_runner.go:840, then inspect how the backup/2TB/n10cpu4 test uses cluster.PutE and reports ssh_problem. Review the linked TeamCity artifacts and Grafana data to determine the cause of the broken scp connection; done means the recurring failure is understood and an appropriate fix or mitigation is identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, go
Domain
infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.