Add retries to release manifests fetch step
- Dominant language
- Go
- Stars
- 2.1k
- Forks
- 328
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 9
Description
**What happened**:
Attempted to upgrade and immediately the operation failed with
```
cluster.go:686: Running shell command [ bin/v0.12.1/eksctl-anywhere upgrade cluster -f eksa-test-8259e41/cluster.yaml -v 4 --kubeconfig long-lived-drib/long-lived-drib-eks-a-cluster.kubeconfig ]
2022-11-22T10:55:47.678-0600 V4 Logger init completed {"vlevel": 4}
2022-11-22T10:55:47.681-0600 V0 Warning: The recommended size of an external etcd cluster is 3 or 5
2022-11-22T10:55:48.984-0600 V0 Warning: The recommended size of an external etcd cluster is 3 or 5
2022-11-22T10:55:52.221-0600 V4 Reading bundles manifest {"url": "https://anywhere-assets.eks.amazonaws.com/releases/bundles/20/manifest.yaml"}
Error: failed to upgrade cluster: unable to get cluster config from file: failed reading file from url [https://distro.eks.amazonaws.com/kubernetes-1-21/kubernetes-1-21-eks-20.yaml]: Get "https://distro.eks.amazonaws.com/kubernetes-1-21/kubernetes-1-21-eks-20.yaml": dial tcp: lookup distro.eks.amazonaws.com on 10.10.0.2:53: read udp 192.168.252.2:0->10.10.0.2:53: i/o timeout
cluster.go:723: Command bin/v0.12.1/eksctl-anywhere [upgrade cluster -f eksa-test-8259e41/cluster.yaml -v 4 --kubeconfig long-lived-drib/long-lived-drib-eks-a-cluster.kubeconfig] failed with error: exit status 255: Error: failed to upgrade cluster: unable to get cluster config from file: failed reading file from url [https://distro.eks.amazonaws.com/kubernetes-1-21/kubernetes-1-21-eks-20.yaml]: Get "https://distro.eks.amazonaws.com/kubernetes-1-21/kubernetes-1-21-eks-20.yaml": dial tcp: lookup distro.eks.amazonaws.com on 10.10.0.2:53: read udp 192.168.252.2:0->10.10.0.2:53: i/o timeout
```
and similarly on cluster deletion
```
2022-11-22T11:56:49.577-0600 V0 Warning: The recommended size of an external etcd cluster is 3 or 5
Error: failed to delete cluster: unable to get cluster config from file: unable to get cluster config from file: failed reading file from url [https://anywhere-assets.eks.amazonaws.com/releases/eks-a/manifest.yaml]: Get "https://anywhere-assets.eks.amazonaws.com/releases/eks-a/manifest.yaml": dial tcp: lookup anywhere-assets.eks.amazonaws.com on 10.10.0.2:53: read udp 192.168.252.2:0->10.10.0.2:53: i/o timeout
```
Which is very disruptive when running operations sequentially and expecting them to succeed
**What you expected to happen**:
Automatic retry when reading from a web server and some connection issue occurs
**How to reproduce it (as minimally and precisely as possible)**:
Difficult to reproduce - it seems nondeterministic
**Anything else we need to know?**:
**Environment**:
- EKS Anywhere Release: v0.12.1 with CloudStack provider e2e test
- EKS Distro Release:
Contributor guide
Assessment
This issue has not been assessed yet.