cockroachdb / cockroachdb/cockroach

basalt-26.3: roachtest: ssh_problem failed

Open
#175,027 60 comments 0 reactions 0 assignees View on GitHub
basalt-green branch-basalt-26.3 O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.ssh_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_GreenRoachtestNightlyGcePrivateBasaltBazel/21511478?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_GreenRoachtestNightlyGcePrivateBasaltBazel/21511478?buildTab=artifacts#/backup-restore/round-trip) on basalt-26.3 @ [9d68ab785115d04ed7ab69c88e35b730f63624e7](https://github.com/cockroachdb/cockroach/commits/9d68ab785115d04ed7ab69c88e35b730f63624e7):

```
test backup-restore/round-trip failed: (test_runner.go:1087).func4: creating Basalt backing cluster teamcity-21511478-1789107039-23-n4cpu4-basalt: not all nodes booted successfully: timed out after 5m: TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]
```

Parameters:
- MetamorphicBasaltMultiAZ=false
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- runtimeAssertionsBuild=true
- storageNodesVersion=v0.1.19-nightly.20260910
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*ssh_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Contributor guide

Open the contributing guide

Research direction

Start with the roachtest README and test_runner.go:1087, then inspect the TeamCity log and backup-restore/round-trip artifacts for build 21511478. Determine why the Basalt backing cluster hit ssh_problem while booting and verify that the roachtest no longer reports this failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, go
Domain
cloud, infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.