cockroachdb / cockroachdb/cockroach

roachtest: ssh_problem failed

Open
#148,191 3 comments 0 reactions 0 assignees View on GitHub
branch-release-23.2.27-rc O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.ssh_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/19911166?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/19911166?buildTab=artifacts#/decommissionBench/nodes=4/warehouses=1000/duration=1h0m0s) on release-23.2.27-rc @ [0a311979123b1628c6389d2d74a62aa19640bdef](https://github.com/cockroachdb/cockroach/commits/0a311979123b1628c6389d2d74a62aa19640bdef):

```
test decommissionBench/nodes=4/warehouses=1000/duration=1h0m0s failed: (test_runner.go:752).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 ubuntu@35.185.7.163:./cockroach
Warning: Permanently added '35.185.7.163' (ECDSA) to the list of known hosts.
: signal: killed [owner=test-eng]
```

Parameters:
- ROACHTEST_arch=amd64
- ROACHTEST_cloud=gce
- ROACHTEST_coverageBuild=false
- ROACHTEST_cpu=16
- ROACHTEST_encrypted=false
- ROACHTEST_fs=ext4
- ROACHTEST_localSSD=true
- ROACHTEST_metamorphicBuild=false
- ROACHTEST_ssd=0
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-19911166/decommissionbench-nodes-4-warehouses-1000-duration-1h0m0s/-62135596800000/-62135596680000)

Same failure on other branches

- #148064 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.1.8-rc]
- #147561 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]
- #147291 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
- #146861 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.1.7-rc]
- #146783 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.26-rc]
- #146715 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.14-rc]
- #146182 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.1.6-rc]
- #146027 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
- #145029 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.18-rc]
- #144757 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.17-rc]

/cc @cockroachdb/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*ssh_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

Jira issue: CRDB-51464

Contributor guide

Open the contributing guide

Research direction

Start with the linked TeamCity failure log and artifacts, then read the roachtest README and the linked Grafana view. Compare this ssh_problem failure with the other linked branch failures and the internal investigation guide. Done means identifying and addressing the recurring SSH or test-infrastructure failure, with the affected roachtest no longer producing this failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, go
Domain
infrastructure, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.