cockroachdb / cockroachdb/cockroach
release-26.3: roachtest: ssh_problem failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
roachtest.ssh_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21414745?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21414745?buildTab=artifacts#/ycsb/D/nodes=3/cpu=32/isolation-level=read-committed) on release-26.3 @ [b96b2082fca995b6a78f8563ed7ba31a9f10bd54](https://github.com/cockroachdb/cockroach/commits/b96b2082fca995b6a78f8563ed7ba31a9f10bd54):
Failed with:
```
test ycsb/D/nodes=3/cpu=32/isolation-level=read-committed failed: (cluster.go:2700).Start: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/ycsb/D/nodes=3/cpu=32/isolation-level=read-committed/run_1/ssh/ssh_182848.030910301_n1_init-cluster.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/ycsb/D/nodes=3/cpu=32/isolation-level=read-committed/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21414745-1783491255-250-n4cpu32-0001 | 34.23.0.204 | 10.142.0.19 |
| teamcity-21414745-1783491255-250-n4cpu32-0002 | 34.138.42.109 | 10.142.0.61 |
| teamcity-21414745-1783491255-250-n4cpu32-0003 | 34.139.15.245 | 10.142.0.20 |
| teamcity-21414745-1783491255-250-n4cpu32-0004 | 35.231.153.183 | 10.142.0.64 |
Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=32
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- runtimeAssertionsBuild=false
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21414745/ycsb-d-nodes-3-cpu-32-isolation-level-read-committed/1783535312580/1783535590090)
See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21414745-1783491255-250-n4cpu32&from_ts=1783535312580&to_ts=1783535590090&live=false&storage=flex_tier)
/cc @cockroachlabs/test-eng
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*ssh_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-65598
Contributor guide
Research direction
Start with the retained ssh -vvv log at /artifacts/ycsb/D/nodes=3/cpu=32/isolation-level=read-committed/run_1/ssh/ssh_182848.030910301_n1_init-cluster.log and the failure at cluster.go:2700. Read pkg/cmd/roachtest/README.md and compare the roachdash or linked build artifacts. Done means identifying the SSH flake cause and preventing this roachtest failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, go
- Domain
- infrastructure, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100