aws / aws/eks-anywhere

SnowBallEdge EKS-A Upgrade fails for Instance Type Upgrade.

Open
#5,812 3 comments 1 reaction 0 assignees View on GitHub
area/updates
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**:

SnowBallEdge EKS-A Upgrade fails for Instance Type Upgrade. I tried to upgrade from `large` instance to `2xlarge` instance for nodes and the upgrade fails with below errors

```
2023-05-09T15:53:23.575Z V6 Executing command {"cmd": "/usr/bin/docker rm -f -v eksa_1683647549800094637"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x110 pc=0x2a2604c]

goroutine 1 [running]:
github.com/aws/eks-anywhere/pkg/providers/snow.(*SnowProvider).EnvMap(0x2dc0e40?, 0xc000570f80)
github.com/aws/eks-anywhere/pkg/providers/snow/snow.go:197 +0x4c
github.com/aws/eks-anywhere/pkg/executables.(*Clusterctl).InstallEtcdadmProviders(0xc00037bc40, {0x3a65990, 0xc000126000}, 0x1c?, 0xc00041a330, {0x3a73eb0, 0xc0002eb860?}, {0xc0001415e0, 0x2, 0x2})
github.com/aws/eks-anywhere/pkg/executables/clusterctl.go:422 +0x2df
github.com/aws/eks-anywhere/pkg/clusterapi.(*Installer).EnsureEtcdProvidersInstallation(0xc00056e618, {0x3a65990, 0xc000126000}, 0xc00041a330, {0x3a73eb0, 0xc0002eb860}, 0xc000570f80)
github.com/aws/eks-anywhere/pkg/clusterapi/installer.go:49 +0x2bc
github.com/aws/eks-anywhere/pkg/workflows.(*ensureEtcdCAPIComponentsExistTask).Run(0xc000548e80?, {0x3a65990, 0xc000126000}, 0xc0001320f0)
github.com/aws/eks-anywhere/pkg/workflows/upgrade.go:206 +0x96
github.com/aws/eks-anywhere/pkg/task.(*taskRunner).RunTask(0xc00041a360, {0x3a65990, 0xc000126000}, 0xc0001320f0)
github.com/aws/eks-anywhere/pkg/task/task.go:155 +0x3e3
github.com/aws/eks-anywhere/pkg/workflows.(*Upgrade).Run(0xc000adfc80, {0x3a65990, 0xc000126000}, 0xc0002edd80, 0xc00041a330, 0xc00041a330, {0x3a32d80?, 0xc00056e640}, 0x0?)
github.com/aws/eks-anywhere/pkg/workflows/upgrade.go:78 +0x435
github.com/aws/eks-anywhere/cmd/eksctl-anywhere/cmd.(*upgradeClusterOptions).upgradeCluster(0x52758c0, 0x5253c00)
github.com/aws/eks-anywhere/cmd/eksctl-anywhere/cmd/upgradecluster.go:149 +0xa98
github.com/aws/eks-anywhere/cmd/eksctl-anywhere/cmd.glob..func26(0x5253c00?, {0x34ec5e9?, 0x3?, 0x3?})
github.com/aws/eks-anywhere/cmd/eksctl-anywhere/cmd/upgradecluster.go:37 +0x26
github.com/spf13/cobra.(*Command).execute(0x5253c00, {0xc00058c360, 0x3, 0x3})
github.com/spf13/cobra@v1.5.0/command.go:872 +0x694
github.com/spf13/cobra.(*Command).ExecuteC(0x5253200)
github.com/spf13/cobra@v1.5.0/command.go:990 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
github.com/spf13/cobra@v1.5.0/command.go:918
github.com/spf13/cobra.(*Command).ExecuteContext(...)
github.com/spf13/cobra@v1.5.0/command.go:911
github.com/aws/eks-anywhere/cmd/eksctl-anywhere/cmd.Execute()
github.com/aws/eks-anywhere/cmd/eksctl-anywhere/cmd/root.go:59 +0x54
main.main()
github.com/aws/eks-anywhere/cmd/eksctl-anywhere/main.go:29 +0x125
```

**What you expected to happen**:

Upgrade of worker nodes to `2xlarge` instance.

**How to reproduce it (as minimally and precisely as possible)**:

Create an EKS-A Cluster on Snow with `large` instance of 3 node size for CP and DP and try to upgrade the instance type to `2xlarge` for workers alone.

**Anything else we need to know?**:

**Environment**:
- EKS Anywhere Release: v0.14.1
- EKS Distro Release:

Contributor guide

Open the contributing guide

Research direction

Start at pkg/providers/snow/snow.go:197 in EnvMap, then follow the upgrade path through pkg/executables/clusterctl.go and pkg/workflows/upgrade.go shown in the stack trace. Reproduce the Snowball Edge worker instance-type upgrade and trace the nil value; done means upgrading from large to 2xlarge completes without the panic.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.