cockroachdb / cockroachdb/cockroach

release-26.3.2-rc: roachtest: ssh_problem failed

Open
#174,722 30 comments 0 reactions 0 assignees View on GitHub
branch-release-26.3.2-rc O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.ssh_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21499576?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21499576?buildTab=artifacts#/multi-store-remove) on release-26.3.2-rc @ [6149d98a3244ba2a3e4417d9e363da83e213e1b7](https://github.com/cockroachdb/cockroach/commits/6149d98a3244ba2a3e4417d9e363da83e213e1b7):

```
test multi-store-remove failed: (test_runner.go:1027).func4: not all nodes booted successfully: timed out after 5m: TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]
```

Parameters:
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=2
- runtimeAssertionsBuild=true
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #174706 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.24-rc]
- #174702 release-26.2.7-rc: roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.2.7-rc]
- #174690 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.4.17-rc]
- #173564 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.4.15-rc]
- #173555 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.23-rc]
- #173325 release-26.2.6-rc: roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.2.6-rc]
- #173324 release-26.2.6-rc: roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.2.6-rc]
- #173320 basalt-26.3: roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-basalt-26.3]
- #173321 basalt-26.3: roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-basalt-26.3]
- #173249 release-26.3.1-rc: roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.3.1-rc]

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*ssh_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-67933

Contributor guide

Open the contributing guide

Research direction

Start with the TeamCity log and artifacts for the multi-store-remove failure, then read pkg/cmd/roachtest/README.md for roachtest investigation guidance. Compare the repeated ssh_problem failures listed in this issue and determine the cause of the node boot timeout; done means the failure is addressed and the relevant roachtest no longer reports this problem.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.