cockroachdb / cockroachdb/cockroach
release-26.2.5-rc: roachtest: ssh_problem failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
roachtest.ssh_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21422073?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21422073?buildTab=artifacts#/cdc/mixed-version/checkpointing) on release-26.2.5-rc @ [aecc5c2a65078594fed36cb70bdbf2b0a0aaaf96](https://github.com/cockroachdb/cockroach/commits/aecc5c2a65078594fed36cb70bdbf2b0a0aaaf96):
Failed with:
```
test cdc/mixed-version/checkpointing failed: (mixed_version_cdc.go:741).runCDCMixedVersionCheckpointing: mixed-version test failure while running step 37 (restart system server on node 2 with binary version v26.2.5-dev-aecc5c2a65078594fed36cb70bdbf2b0a0aaaf96 (release-26.2.5-rc)): cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/mixed-version/checkpointing/run_1/ssh/ssh_141753.001288341_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:4503).stop: context canceled
test artifacts and logs in: /artifacts/cdc/mixed-version/checkpointing/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21422073-1784031077-56-n5cpu4-0001 | 136.107.82.14 | 10.150.0.80 |
| teamcity-21422073-1784031077-56-n5cpu4-0002 | 34.48.35.55 | 10.150.0.79 |
| teamcity-21422073-1784031077-56-n5cpu4-0003 | 35.199.39.188 | 10.150.0.81 |
| teamcity-21422073-1784031077-56-n5cpu4-0004 | 34.150.169.46 | 10.150.0.83 |
| teamcity-21422073-1784031077-56-n5cpu4-0005 | 34.85.241.110 | 10.150.0.88 |
Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- mvtDeploymentMode=separate-process
- mvtVersions=v25.4.10 → v26.2.5-dev-aecc5c2a65078594fed36cb70bdbf2b0a0aaaf96 (release-26.2.5-rc)
- runtimeAssertionsBuild=false
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21422073/cdc-mixed-version-checkpointing/1784037365964/1784038864644)
Same failure on other branches
- #172364 release-26.3: roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.3]
- #172363 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #172359 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #171791 release-26.3: roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.3]
- #171649 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.20-rc]
- #171630 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.1.6-rc]
- #171625 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.4.12-rc]
- #171591 release-26.2.3-rc: roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.2.3-rc]
- #171529 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.30-rc]
- #171098 release-26.2.2-rc: roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.2.2-rc]
/cc @cockroachlabs/test-eng
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*ssh_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-65761
Contributor guide
Research direction
Start with the retained SSH log at /artifacts/cdc/mixed-version/checkpointing/run_1/ssh/ssh_141753.001288341_n2_kill.log and the roachtest README in pkg/cmd/roachtest/README.md. Compare the related ssh_problem reports and the failure in mixed_version_cdc.go:741; done means identifying a reproducible cause or confirming the infrastructure flake and defining the appropriate follow-up.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, go
- Domain
- infrastructure, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100