cockroachdb / cockroachdb/cockroach

release-26.3: roachtest: ssh_problem failed

Open
#172,364 128 comments 0 reactions 0 assignees View on GitHub
branch-release-26.3 O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.ssh_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21414745?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21414745?buildTab=artifacts#/ycsb/D/nodes=3/cpu=32/isolation-level=read-committed) on release-26.3 @ [b96b2082fca995b6a78f8563ed7ba31a9f10bd54](https://github.com/cockroachdb/cockroach/commits/b96b2082fca995b6a78f8563ed7ba31a9f10bd54):

Failed with:

```
test ycsb/D/nodes=3/cpu=32/isolation-level=read-committed failed: (cluster.go:2700).Start: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/ycsb/D/nodes=3/cpu=32/isolation-level=read-committed/run_1/ssh/ssh_182848.030910301_n1_init-cluster.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/ycsb/D/nodes=3/cpu=32/isolation-level=read-committed/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21414745-1783491255-250-n4cpu32-0001 | 34.23.0.204 | 10.142.0.19 |
| teamcity-21414745-1783491255-250-n4cpu32-0002 | 34.138.42.109 | 10.142.0.61 |
| teamcity-21414745-1783491255-250-n4cpu32-0003 | 34.139.15.245 | 10.142.0.20 |
| teamcity-21414745-1783491255-250-n4cpu32-0004 | 35.231.153.183 | 10.142.0.64 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=32
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21414745/ycsb-d-nodes-3-cpu-32-isolation-level-read-committed/1783535312580/1783535590090)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21414745-1783491255-250-n4cpu32&from_ts=1783535312580&to_ts=1783535590090&live=false&storage=flex_tier)

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*ssh_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-65598

Contributor guide

Open the contributing guide

Research direction

Start with the retained ssh -vvv log at /artifacts/ycsb/D/nodes=3/cpu=32/isolation-level=read-committed/run_1/ssh/ssh_182848.030910301_n1_init-cluster.log and the failure at cluster.go:2700. Read pkg/cmd/roachtest/README.md and compare the roachdash or linked build artifacts. Done means identifying the SSH flake cause and preventing this roachtest failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, go
Domain
infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.