cockroachdb / cockroachdb/cockroach

roachtest: ssh_problem failed

Open
#173,564 10 comments 0 reactions 0 assignees View on GitHub
branch-release-25.4.15-rc O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.ssh_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAwsBazel/21473010?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAwsBazel/21473010?buildTab=artifacts#/ycsb/E/nodes=3/cpu=32) on release-25.4.15-rc @ [4913a926920d3003e1223156615a6a6a7d524b59](https://github.com/cockroachdb/cockroach/commits/4913a926920d3003e1223156615a6a6a7d524b59):

```
test ycsb/E/nodes=3/cpu=32 failed: (test_runner.go:901).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: error persisted after 3 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 ubuntu@3.133.85.182:./cockroach
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]
```

Parameters:
- arch=arm64
- cloud=aws
- coverageBuild=false
- cpu=32
- encrypted=false
- runtimeAssertionsBuild=false
- ssd=0
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

_Grafana is not yet available for aws clusters_

Same failure on other branches

- #173555 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.23-rc]
- #172557 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.4.14-rc]
- #172517 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.22-rc]
- #172515 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.1.8-rc]

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*ssh_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Contributor guide

Open the contributing guide

Research direction

Start with the roachtest README and the failure at test_runner.go:901, then inspect the linked TeamCity log and artifacts for the AWS arm64 ycsb/E run. Investigate why the scp transfer ends with a broken pipe across the listed branches; done means the ssh_problem failure is understood and the affected roachtest behavior is fixed or reliably diagnosed.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
infrastructure, networking, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.