cockroachdb / cockroachdb/cockroach

release-26.3: roachtest: ssh_problem failed

Open
#171,791 23 comments 0 reactions 0 assignees View on GitHub
branch-release-26.3 O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.ssh_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21392022?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21392022?buildTab=artifacts#/inspect/throughput/bulkingest/nodes=12/cpu=8/rows=500000000/checks=1/unique/complex=false) on release-26.3 @ [8a9a8e3305daa0890b7b083c13cb572f911bd6cc](https://github.com/cockroachdb/cockroach/commits/8a9a8e3305daa0890b7b083c13cb572f911bd6cc):

```
test inspect/throughput/bulkingest/nodes=12/cpu=8/rows=500000000/checks=1/unique/complex=false failed: (test_runner.go:1017).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -R -A -i /home/roach/.ssh/id_rsa ubuntu@35.243.250.75:./cockroach ubuntu@34.148.65.24:./cockroach
ssh: connect to host 34.148.65.24 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]
```

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=8
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21392022/inspect-throughput-bulkingest-nodes-12-cpu-8-rows-500000000-che/-62135596800000/-62135596680000)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21392022-1781763104-188-n13cpu8-geo&from_ts=-62135596800000&to_ts=-62135596680000&live=false&storage=flex_tier)

/cc @cockroachdb/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*ssh_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-64927

Contributor guide

Open the contributing guide

Research direction

Start with pkg/cmd/roachtest/README.md and test_runner.go around line 1017, then review the linked TeamCity artifacts, Grafana dashboard, and Datadog logs. Determine why the GCE roachtest encountered the SSH timeout and verify the relevant test completes successfully without the ssh_problem failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, go
Domain
cloud, infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.