cockroachdb / cockroachdb/cockroach

roachtest: ssh_problem failed

Open
#172,359 19 comments 0 reactions 0 assignees View on GitHub
branch-master O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.ssh_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_UpdatePgoProfile/21415558?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_UpdatePgoProfile/21415558?buildTab=artifacts#/tpcc-nowait/literal/w=1000/nodes=5/cpu=16) on master @ [b2fdb000dc205ced01163bb650fc72c91856c5cf](https://github.com/cockroachdb/cockroach/commits/b2fdb000dc205ced01163bb650fc72c91856c5cf):

Failed with:

```
test tpcc-nowait/literal/w=1000/nodes=5/cpu=16 failed: (cluster.go:2700).Start: checking cluster bootstrap state: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpcc-nowait/literal/w=1000/nodes=5/cpu=16/run_5/ssh/ssh_181639.926572429_n1_check-bootstrapped.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpcc-nowait/literal/w=1000/nodes=5/cpu=16/run_5
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21415558-1783534384-02-n6cpu16-0001 | 35.196.135.49 | 10.142.0.174 |
| teamcity-21415558-1783534384-02-n6cpu16-0002 | 34.75.89.194 | 10.142.0.141 |
| teamcity-21415558-1783534384-02-n6cpu16-0003 | 35.227.9.182 | 10.142.0.169 |
| teamcity-21415558-1783534384-02-n6cpu16-0004 | 34.26.127.241 | 10.142.0.172 |
| teamcity-21415558-1783534384-02-n6cpu16-0005 | 34.148.100.188 | 10.142.0.165 |
| teamcity-21415558-1783534384-02-n6cpu16-0006 | 34.74.250.17 | 10.142.1.47 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=16
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21415558/tpcc-nowait-literal-w-1000-nodes-5-cpu-16/1783534567855/1783534796087)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21415558-1783534384-02-n6cpu16&from_ts=1783534567855&to_ts=1783534796087&live=false&storage=flex_tier)

Same failure on other branches

- #171649 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.20-rc]
- #171630 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.1.6-rc]
- #171625 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.4.12-rc]
- #171529 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.30-rc]
- #170436 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.1.5-rc]
- #170428 roachtest: ssh_problem failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.4.11-rc]
- #170380 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.19-rc]

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*ssh_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-65592

Contributor guide

Open the contributing guide

Research direction

Start at cluster.go:2700 and inspect the retained SSH log at /artifacts/tpcc-nowait/literal/w=1000/nodes=5/cpu=16/run_5/ssh/ssh_181639.926572429_n1_check-bootstrapped.log. Read the roachtest README, compare the related ssh_problem issues, and use the linked Grafana and Datadog views. Done should include an identified cause with evidence from reproduction or subsequent roachtest runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, google-cloud
Domain
distributed-systems, infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.