cockroachdb / cockroachdb/cockroach

roachprod: investigate SSH issues with AWS on CLI setup-ssh

Open
#115,696 1 comment 0 reactions 0 assignees View on GitHub
A-testeng-infra A-testing C-investigation T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Describe the problem:

For some roachprod users `setup-ssh` with an AWS cluster does not always seem to work even if the local configuration of the machine is correct:

```
# roachprod setup-ssh adityamaru-scale-rbx-test
...
16:50:08 cluster_synced.go:1405: 1: timed out after 5m: SSH_PROBLEM: exit status 255
adityamaru-scale-rbx-test: waiting for nodes to start 0/14 \
16:50:39 cluster_synced.go:1405: 3: timed out after 5m: SSH_PROBLEM: exit status 255
adityamaru-scale-rbx-test: waiting for nodes to start 0/14 \
16:50:49 cluster_synced.go:1405: 2: timed out after 5m: SSH_PROBLEM: exit status 255
adityamaru-scale-rbx-test: waiting for nodes to start 0/14 -
16:58:16 cluster_synced.go:1405: 13: timed out after 5m: SSH_PROBLEM: exit status 255
16:58:16 cluster_synced.go:174: encountered [timed out after 5m: SSH_PROBLEM: exit status 255] on attempt 3 of 3
Error: UNCLASSIFIED_PROBLEM: not all nodes booted successfully
(1) UNCLASSIFIED_PROBLEM
Wraps: (2) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Wait
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:1414
| github.com/cockroachdb/cockroach/pkg/roachprod.SetupSSH
| github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:628
| main.glob..func2
| main/pkg/cmd/roachprod/main.go:156
| main.init.wrap.func4
| main/pkg/cmd/roachprod/main.go:76
| github.com/spf13/cobra.(*Command).execute
| github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:860
| github.com/spf13/cobra.(*Command).ExecuteC
| github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:974
| github.com/spf13/cobra.(*Command).Execute
| github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:902
| main.main
| main/pkg/cmd/roachprod/main.go:1533
| runtime.main
| GOROOT/src/runtime/proc.go:267
| runtime.goexit
| src/runtime/asm_arm64.s:1197
Wraps: (3) not all nodes booted successfully
Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
```

**To Reproduce**

Unfortunately there is no clear way to reproduce this and might be dependent on various steps that `setup-ssh` goes through. A review of the AWS ssh configuration code might lead to an answer.

Jira issue: CRDB-34155

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.