cockroachdb / cockroachdb/cockroach

roachtest: ssh_problem failed

Open
#155,684 5 comments 0 reactions 0 assignees View on GitHub
branch-release-25.4.0-rc O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.ssh_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/20621956?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/20621956?buildTab=artifacts#/failover/chaos/read-only) on release-25.4.0-rc @ [1de9c4bc217dca385a4f912dbdf828bc5629711a](https://github.com/cockroachdb/cockroach/commits/1de9c4bc217dca385a4f912dbdf828bc5629711a):

```
test failover/chaos/read-only failed: (test_runner.go:873).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 ubuntu@35.237.155.195:./cockroach
ssh: connect to host 35.237.155.195 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]
```

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=2
- encrypted=false
- fs=ext4
- localSSD=false
- runtimeAssertionsBuild=false
- ssd=0
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-20621956/failover-chaos-read-only/-62135596800000/-62135596680000)

Same failure on other branches

- #155430 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.7-rc]
- #155142 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.3.3-rc]
- #154854 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.25-rc]
- #154612 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.21-rc]
- #154528 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.4]
- #153154 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.24-rc]
- #153147 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.20-rc]
- #153140 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.6-rc]
- #152951 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.3.2-rc]
- #152321 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.19-rc]

/cc @cockroachdb/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*ssh_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-55546

Contributor guide

Open the contributing guide

Research direction

Start with the failure at test_runner.go:873 and the failover/chaos/read-only artifacts from TeamCity build 20621956. Read pkg/cmd/roachtest/README.md and compare the related ssh_problem reports to determine whether the GCE SSH timeout is reproducible and what reliable test behavior should be restored.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, go
Domain
cloud, infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.