cockroachdb / cockroachdb/cockroach
roachtest: ssh_problem failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
roachtest.ssh_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_UpdatePgoProfile/21415558?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_UpdatePgoProfile/21415558?buildTab=artifacts#/tpcc-nowait/literal/w=1000/nodes=5/cpu=16) on master @ [b2fdb000dc205ced01163bb650fc72c91856c5cf](https://github.com/cockroachdb/cockroach/commits/b2fdb000dc205ced01163bb650fc72c91856c5cf):
Failed with:
```
test tpcc-nowait/literal/w=1000/nodes=5/cpu=16 failed: (cluster.go:2700).Start: checking cluster bootstrap state: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpcc-nowait/literal/w=1000/nodes=5/cpu=16/run_5/ssh/ssh_181639.926572429_n1_check-bootstrapped.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpcc-nowait/literal/w=1000/nodes=5/cpu=16/run_5
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21415558-1783534384-02-n6cpu16-0001 | 35.196.135.49 | 10.142.0.174 |
| teamcity-21415558-1783534384-02-n6cpu16-0002 | 34.75.89.194 | 10.142.0.141 |
| teamcity-21415558-1783534384-02-n6cpu16-0003 | 35.227.9.182 | 10.142.0.169 |
| teamcity-21415558-1783534384-02-n6cpu16-0004 | 34.26.127.241 | 10.142.0.172 |
| teamcity-21415558-1783534384-02-n6cpu16-0005 | 34.148.100.188 | 10.142.0.165 |
| teamcity-21415558-1783534384-02-n6cpu16-0006 | 34.74.250.17 | 10.142.1.47 |
Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=16
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- runtimeAssertionsBuild=false
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21415558/tpcc-nowait-literal-w-1000-nodes-5-cpu-16/1783534567855/1783534796087)
See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21415558-1783534384-02-n6cpu16&from_ts=1783534567855&to_ts=1783534796087&live=false&storage=flex_tier)
Same failure on other branches
- #171649 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.20-rc]
- #171630 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.1.6-rc]
- #171625 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.4.12-rc]
- #171529 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.30-rc]
- #170436 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.1.5-rc]
- #170428 roachtest: ssh_problem failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.4.11-rc]
- #170380 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.19-rc]
/cc @cockroachlabs/test-eng
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*ssh_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-65592
Contributor guide
Research direction
Start at cluster.go:2700 and inspect the retained SSH log at /artifacts/tpcc-nowait/literal/w=1000/nodes=5/cpu=16/run_5/ssh/ssh_181639.926572429_n1_check-bootstrapped.log. Read the roachtest README, compare the related ssh_problem issues, and use the linked Grafana and Datadog views. Done should include an identified cause with evidence from reproduction or subsequent roachtest runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, google-cloud
- Domain
- distributed-systems, infrastructure, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100