cockroachdb / cockroachdb/cockroach
roachprod: investigate SSH issues with AWS on CLI setup-ssh
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Describe the problem:
For some roachprod users `setup-ssh` with an AWS cluster does not always seem to work even if the local configuration of the machine is correct:
```
# roachprod setup-ssh adityamaru-scale-rbx-test
...
16:50:08 cluster_synced.go:1405: 1: timed out after 5m: SSH_PROBLEM: exit status 255
adityamaru-scale-rbx-test: waiting for nodes to start 0/14 \
16:50:39 cluster_synced.go:1405: 3: timed out after 5m: SSH_PROBLEM: exit status 255
adityamaru-scale-rbx-test: waiting for nodes to start 0/14 \
16:50:49 cluster_synced.go:1405: 2: timed out after 5m: SSH_PROBLEM: exit status 255
adityamaru-scale-rbx-test: waiting for nodes to start 0/14 -
16:58:16 cluster_synced.go:1405: 13: timed out after 5m: SSH_PROBLEM: exit status 255
16:58:16 cluster_synced.go:174: encountered [timed out after 5m: SSH_PROBLEM: exit status 255] on attempt 3 of 3
Error: UNCLASSIFIED_PROBLEM: not all nodes booted successfully
(1) UNCLASSIFIED_PROBLEM
Wraps: (2) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Wait
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:1414
| github.com/cockroachdb/cockroach/pkg/roachprod.SetupSSH
| github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:628
| main.glob..func2
| main/pkg/cmd/roachprod/main.go:156
| main.init.wrap.func4
| main/pkg/cmd/roachprod/main.go:76
| github.com/spf13/cobra.(*Command).execute
| github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:860
| github.com/spf13/cobra.(*Command).ExecuteC
| github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:974
| github.com/spf13/cobra.(*Command).Execute
| github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:902
| main.main
| main/pkg/cmd/roachprod/main.go:1533
| runtime.main
| GOROOT/src/runtime/proc.go:267
| runtime.goexit
| src/runtime/asm_arm64.s:1197
Wraps: (3) not all nodes booted successfully
Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
```
**To Reproduce**
Unfortunately there is no clear way to reproduce this and might be dependent on various steps that `setup-ssh` goes through. A review of the AWS ssh configuration code might lead to an answer.
Jira issue: CRDB-34155
Contributor guide
Assessment
This issue has not been assessed yet.