aws / aws/eks-anywhere

Add retries to release manifests fetch step

Open
#4,194 1 comment 0 reactions 1 assignee Claimed by @annaveretennykova View on GitHub
area/cli good first issue team/cli
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**:
Attempted to upgrade and immediately the operation failed with
```
cluster.go:686: Running shell command [ bin/v0.12.1/eksctl-anywhere upgrade cluster -f eksa-test-8259e41/cluster.yaml -v 4 --kubeconfig long-lived-drib/long-lived-drib-eks-a-cluster.kubeconfig ]
2022-11-22T10:55:47.678-0600 V4 Logger init completed {"vlevel": 4}
2022-11-22T10:55:47.681-0600 V0 Warning: The recommended size of an external etcd cluster is 3 or 5
2022-11-22T10:55:48.984-0600 V0 Warning: The recommended size of an external etcd cluster is 3 or 5
2022-11-22T10:55:52.221-0600 V4 Reading bundles manifest {"url": "https://anywhere-assets.eks.amazonaws.com/releases/bundles/20/manifest.yaml"}
Error: failed to upgrade cluster: unable to get cluster config from file: failed reading file from url [https://distro.eks.amazonaws.com/kubernetes-1-21/kubernetes-1-21-eks-20.yaml]: Get "https://distro.eks.amazonaws.com/kubernetes-1-21/kubernetes-1-21-eks-20.yaml": dial tcp: lookup distro.eks.amazonaws.com on 10.10.0.2:53: read udp 192.168.252.2:0->10.10.0.2:53: i/o timeout
cluster.go:723: Command bin/v0.12.1/eksctl-anywhere [upgrade cluster -f eksa-test-8259e41/cluster.yaml -v 4 --kubeconfig long-lived-drib/long-lived-drib-eks-a-cluster.kubeconfig] failed with error: exit status 255: Error: failed to upgrade cluster: unable to get cluster config from file: failed reading file from url [https://distro.eks.amazonaws.com/kubernetes-1-21/kubernetes-1-21-eks-20.yaml]: Get "https://distro.eks.amazonaws.com/kubernetes-1-21/kubernetes-1-21-eks-20.yaml": dial tcp: lookup distro.eks.amazonaws.com on 10.10.0.2:53: read udp 192.168.252.2:0->10.10.0.2:53: i/o timeout
```

and similarly on cluster deletion

```
2022-11-22T11:56:49.577-0600 V0 Warning: The recommended size of an external etcd cluster is 3 or 5
Error: failed to delete cluster: unable to get cluster config from file: unable to get cluster config from file: failed reading file from url [https://anywhere-assets.eks.amazonaws.com/releases/eks-a/manifest.yaml]: Get "https://anywhere-assets.eks.amazonaws.com/releases/eks-a/manifest.yaml": dial tcp: lookup anywhere-assets.eks.amazonaws.com on 10.10.0.2:53: read udp 192.168.252.2:0->10.10.0.2:53: i/o timeout
```

Which is very disruptive when running operations sequentially and expecting them to succeed

**What you expected to happen**:
Automatic retry when reading from a web server and some connection issue occurs

**How to reproduce it (as minimally and precisely as possible)**:
Difficult to reproduce - it seems nondeterministic

**Anything else we need to know?**:

**Environment**:
- EKS Anywhere Release: v0.12.1 with CloudStack provider e2e test
- EKS Distro Release:

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.