Cluster creation fails when adding ociNamespace in spec
- Dominant language
- Go
- Stars
- 2.1k
- Forks
- 328
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 9
Description
**What happened**:
We are using a private registry(Nexus) for images/artifacts. We downloaded images/artifacts following [this](https://anywhere.eks.amazonaws.com/docs/reference/clusterspec/optional/registrymirror/#import-images-into-a-private-registry) steps. We could successfully import all the images/artifacts into Nexus. We want to create clusters using resources in Nexus by leveraing Registry Mirror configuration in cluster spec as in below.
```apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
name: my-cluster-name
spec:
...
registryMirrorConfiguration:
endpoint:
port:
caCertContent: |
-----BEGIN CERTIFICATE-----
MIIF1DCCA...
...
es6RXmsCj...
-----END CERTIFICATE-----
```
This is working fine, cluster is coming up and we can install curated packages on workload cluster. However, when we checked installed packages PODs, we saw `ImagePullBackOff` on the PODs and registry was pointing to ECR rather than Nexus private registry. We had to change `Default Image Registry` to Nexus in pbc. After this change all worked fine.
So we did not want to do manual modification, we wanted to add `ociNamespace` to cluster spec as following config to avoid this image pull back off error.
```
spec:
registryMirrorConfiguration:
endpoint: example-private-reg.com
port: 8426
ociNamespaces:
- registry: "public.ecr.aws"
namespace: "eks-anywhere"
- registry: "783794618700.dkr.ecr.us-east-1.amazonaws.com"
namespace: "curated-packages"
caCertContent: |
```
We used following command to spin up the cluster.
```
eksctl anywhere create cluster \
# --install-packages packages.yaml \ # uncomment to install curated packages at cluster creation
-f eksa-mgmt-cluster.yaml
```
This time we hitting an issue with cluster creation and it throws below error.
`2023-03-28T17:18:49.541Z V5 Error happened during retry {"error": "Error response from daemon: manifest for example-private-reg.com:8426/eks-anywhere/eks-anywhere/cli-tools:v0.14.4-eks-a-30 not found: manifest unknown: manifest unknown\n", "retries": 5}
2`
It seems like it is adding an addition `eks-anywhere` path to the registry end point.
**What you expected to happen**:
We want to have right registry configuration in pbc so that POD can work fine. This is possible it by populating private registry endpoint in pbc along with `curated-packages` path.
**How to reproduce it (as minimally and precisely as possible)**:
**Anything else we need to know?**:
**Environment**: vsphere
- EKS Anywhere Release:0.14.4
- EKS Distro Release:
Contributor guide
Research direction
Start with the registryMirrorConfiguration and ociNamespaces sections in eksa-mgmt-cluster.yaml, then run the shown eksctl anywhere create cluster command against the private registry. Use the reported manifest error to trace the registry path handling; done means cluster creation succeeds without duplicating the eks-anywhere namespace and curated package images resolve from the private registry.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100