cockroachdb / cockroachdb/cockroach

roachtest: ssh_problem failed

Open
#150,125 3 comments 0 reactions 0 assignees View on GitHub
branch-release-25.1.9-rc O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.ssh_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/20164999?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/20164999?buildTab=artifacts#/follower-reads/survival=zone/locality=global/reads=exact-staleness) on release-25.1.9-rc @ [da9b0ac6f6220c719b417a4e15d975773a8e74be](https://github.com/cockroachdb/cockroach/commits/da9b0ac6f6220c719b417a4e15d975773a8e74be):

```
test follower-reads/survival=zone/locality=global/reads=exact-staleness failed: (test_runner.go:823).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach-ea.linux-amd64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach-ea.linux-amd64 ubuntu@34.39.21.102:./cockroach
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]
```

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- encrypted=false
- runtimeAssertionsBuild=true
- ssd=0
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-20164999/follower-reads-survival-zone-locality-global-reads-exact-stalen/-62135596800000/-62135596680000)

Same failure on other branches

- #150096 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.21-rc]
- #149972 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.16-rc]
- #149737 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.3-rc]
- #149125 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.3]
- #148346 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.15-rc]
- #148333 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.2-rc]
- #148257 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.20-rc]
- #148191 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.27-rc]
- #148064 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.1.8-rc]
- #147561 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

/cc @cockroachdb/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*ssh_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-52576

Contributor guide

Open the contributing guide

Research direction

Start with pkg/cmd/roachtest/README.md and the failure at test_runner.go:823. Review the linked TeamCity build, artifacts, and Grafana data to trace the ssh_problem during cluster.PutE. Done means identifying the cause and verifying that the affected roachtest no longer fails with this SSH error.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, go
Domain
cloud, infrastructure, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.