cockroachdb / cockroachdb/cockroach

roachtest: ssh_problem failed

Open
#147,291 12 comments 0 reactions 0 assignees View on GitHub
branch-release-25.2.1-rc O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.ssh_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/19780278?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/19780278?buildTab=artifacts#/sql-stats/mixed-version) on release-25.2.1-rc @ [592da43f5cc80f32a8277aa90e09c7e3362ff53c](https://github.com/cockroachdb/cockroach/commits/592da43f5cc80f32a8277aa90e09c7e3362ff53c):

```
test sql-stats/mixed-version failed: (mixedversion.go:829).Run: mixed-version test failure while running step 138 (restart mixed-version-tenant-9fvrs server on node 1 with binary version release-25.2.1-rc): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/sql-stats/mixed-version/run_1/ssh/ssh_113738.286037514_n1_run-start-script.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/sql-stats/mixed-version/run_1
```

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- encrypted=false
- fs=ext4
- localSSD=true
- metamorphicBufferedSender=false
- mvtDeploymentMode=separate-process
- mvtVersions=v23.2.25 → v24.1.18 → v24.2.10 → v24.3.13 → release-25.2.1-rc
- runtimeAssertionsBuild=false
- ssd=0
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-19780278/sql-stats-mixed-version/1748257880542/1748260126335)

Same failure on other branches

- #146861 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.1.7-rc]
- #146783 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.26-rc]
- #146715 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.14-rc]
- #146182 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.1.6-rc]
- #146027 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
- #145029 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.18-rc]
- #144757 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.17-rc]
- #144607 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.24-rc]
- #144541 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.1.5-rc]
- #144529 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.11-rc]

/cc @cockroachdb/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*ssh_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-50980

Contributor guide

Open the contributing guide

Research direction

Start with pkg/cmd/roachtest/README.md and the failure at mixedversion.go:829. Inspect the retained ssh -vvv log in /artifacts/sql-stats/mixed-version/run_1/ssh/ssh_113738.286037514_n1_run-start-script.log, the Grafana run, and the related ssh_problem issues. Done means establishing the recurring SSH failure's cause and documenting or implementing an appropriate resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
infrastructure, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.