aws / aws/eks-anywhere-packages
Breaking changes in latest package bundles for credential-provider-package
- Dominant language
- Go
- Stars
- 31
- Forks
- 36
- PR merge metrics
- No merged PRs in 30d
Description
**What happened**:
Any version of package bundle above v1-27-128 are not usable because of multiple issues. Should we keep those faulty packages in registry? Is there any end2end test that could detect that in future?
```
v1-27-137 other 7 days ago public.ecr.aws/eks-anywhere/...es-bundles:v1-27-137 60.8 KB > doesn't work, image of anywhere-package controller works fine, see log_1
v1-27-134 other 14 days ago public.ecr.aws/eks-anywhere/...es-bundles:v1-27-134 60.7 KB > faulty secret, propably because of helm chart
v1-27-130 other 17 days ago public.ecr.aws/eks-anywhere/...es-bundles:v1-27-130 60.7 KB > wrong helm app version which causing imagepullbackoff for package controler and refresher, see log_2
v1-27-129 other 17 days ago public.ecr.aws/eks-anywhere/...es-bundles:v1-27-129 60.7 KB > wrong helm app version which causing imagepullbackoff for package controler and refresher, see log_2
v1-27-128 other 2 months ago public.ecr.aws/eks-anywhere/...es-bundles:v1-27-128 > works fine
```
log_1
```
2023-11-02T10:52:24.780Z ECRCredInjector Failed to inject ECR credential to docker config {"error": "operation error ECR: GetAuthorizationToken, failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, static credentials are empty"}
github.com/aws/eks-anywhere-packages/pkg/registry.(*ECRCredInjector).Run
github.com/aws/eks-anywhere-packages/pkg/registry/ecr_cred_injector.go:56
```
log_2
```
eksa-packages eks-anywhere-packages 8 failed eks-anywhere-packages-0.0.0-8862036270224f2a6b8d6ecd455b6b1fa1084619 v0.0.0-8862036270224f2a6b8d6ecd455b6b1fa1084619
```
**What you expected to happen**:
eks-anywhere-packages shouldn't be published with such breaking changes
**How to reproduce it (as minimally and precisely as possible)**:
1. Using 0.17.4 eks-anywhere install k8s 1.27 using tinkerbell provider
2. Create eks-anywhere role along with anchor, follow https://anywhere.eks.amazonaws.com/docs/packages/credential-provider-package/iam_roles_anywhere/#prerequisites
3. Create aws-config secret in eks-packages ns
```
[default]
region = eu-west-1
credential_process = aws_signing_helper credential-process --certificate /var/lib/kubelet/pki/kubelet-client-current.pem --private-key /var/lib/kubelet/pki/kubelet-client-current.pem --profile-arn $PROFILE_ARN --role-arn $ROLE_ARN --trust-anchor-arn $TRUST_ANCHOR_ARN
```
4. Add package to download from private ECR registry
```
apiVersion: packages.eks.amazonaws.com/v1alpha1
kind: Package
metadata:
name: my-credential-provider-package
namespace: eksa-packages-eksa
annotations:
"helm.sh/resource-policy": keep
"anywhere.eks.aws.com/internal": "true"
spec:
packageName: credential-provider-package
targetNamespace: eksa-packages
config: |-
tolerations:
- key: "node-role.kubernetes.io/master"
operator: "Exists"
effect: "NoSchedule"
- key: "node-role.kubernetes.io/control-plane"
operator: "Exists"
effect: "NoSchedule"
sourceRegistry: public.ecr.aws/eks-anywhere
credential:
- matchImages:
- 000000000000.dkr.ecr.eu-west-2.amazonaws.com
profile: "default"
secretName: aws-config
defaultCacheDuration: "12h"
```
5. Verify you have latest version of packagebundle in packagebundlecontroller
6. Create pod with image from `000000000000.dkr.ecr.eu-west-2.amazonaws.com` registry
7. ImagePullBackOff should be logged from kubelet
**Anything else we need to know?**:
We also checked latest v1-28 with k8s 1.28 and it also experience issues the same as v1-27-137
**Environment**: k8s 1.27, tinker provisioner with bare bone nodes, ubuntu 22.04 ami
- EKS Anywhere Release: 0.17.4
- EKS Distro Release: -
Contributor guide
Research direction
Start by reproducing the credential-provider-package failures using the listed EKS Anywhere, Kubernetes, AWS configuration, and package bundle versions. Inspect the ECR credential path at pkg/registry/ecr_cred_injector.go:56 and the package bundle or Helm release flow mentioned in the report. Done means faulty bundles are not published and an end-to-end check detects these failures before release.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go, helm, kubernetes
- Domain
- cloud, devops, release, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100