aws / aws/eks-anywhere

Cluster creation fails when adding ociNamespace in spec

Open
#5,479 2 comments 0 reactions 0 assignees View on GitHub
external
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**:
We are using a private registry(Nexus) for images/artifacts. We downloaded images/artifacts following [this](https://anywhere.eks.amazonaws.com/docs/reference/clusterspec/optional/registrymirror/#import-images-into-a-private-registry) steps. We could successfully import all the images/artifacts into Nexus. We want to create clusters using resources in Nexus by leveraing Registry Mirror configuration in cluster spec as in below.
```apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
name: my-cluster-name
spec:
...
registryMirrorConfiguration:
endpoint:
port:
caCertContent: |
-----BEGIN CERTIFICATE-----
MIIF1DCCA...
...
es6RXmsCj...
-----END CERTIFICATE-----
```

This is working fine, cluster is coming up and we can install curated packages on workload cluster. However, when we checked installed packages PODs, we saw `ImagePullBackOff` on the PODs and registry was pointing to ECR rather than Nexus private registry. We had to change `Default Image Registry` to Nexus in pbc. After this change all worked fine.
So we did not want to do manual modification, we wanted to add `ociNamespace` to cluster spec as following config to avoid this image pull back off error.

```
spec:
registryMirrorConfiguration:
endpoint: example-private-reg.com
port: 8426
ociNamespaces:
- registry: "public.ecr.aws"
namespace: "eks-anywhere"
- registry: "783794618700.dkr.ecr.us-east-1.amazonaws.com"
namespace: "curated-packages"
caCertContent: |
```
We used following command to spin up the cluster.
```
eksctl anywhere create cluster \
# --install-packages packages.yaml \ # uncomment to install curated packages at cluster creation
-f eksa-mgmt-cluster.yaml
```
This time we hitting an issue with cluster creation and it throws below error.
`2023-03-28T17:18:49.541Z V5 Error happened during retry {"error": "Error response from daemon: manifest for example-private-reg.com:8426/eks-anywhere/eks-anywhere/cli-tools:v0.14.4-eks-a-30 not found: manifest unknown: manifest unknown\n", "retries": 5}
2`
It seems like it is adding an addition `eks-anywhere` path to the registry end point.

**What you expected to happen**:
We want to have right registry configuration in pbc so that POD can work fine. This is possible it by populating private registry endpoint in pbc along with `curated-packages` path.

**How to reproduce it (as minimally and precisely as possible)**:

**Anything else we need to know?**:

**Environment**: vsphere
- EKS Anywhere Release:0.14.4
- EKS Distro Release:

Contributor guide

Open the contributing guide

Research direction

Start with the registryMirrorConfiguration and ociNamespaces sections in eksa-mgmt-cluster.yaml, then run the shown eksctl anywhere create cluster command against the private registry. Use the reported manifest error to trace the registry path handling; done means cluster creation succeeds without duplicating the eks-anywhere namespace and curated package images resolve from the private registry.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.