aws / aws/eks-anywhere-packages

Breaking changes in latest package bundles for credential-provider-package

Open
#1,024 6 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
31
Forks
36
PR merge metrics
No merged PRs in 30d

Description

**What happened**:
Any version of package bundle above v1-27-128 are not usable because of multiple issues. Should we keep those faulty packages in registry? Is there any end2end test that could detect that in future?
```
v1-27-137 other 7 days ago public.ecr.aws/eks-anywhere/...es-bundles:v1-27-137 60.8 KB > doesn't work, image of anywhere-package controller works fine, see log_1
v1-27-134 other 14 days ago public.ecr.aws/eks-anywhere/...es-bundles:v1-27-134 60.7 KB > faulty secret, propably because of helm chart
v1-27-130 other 17 days ago public.ecr.aws/eks-anywhere/...es-bundles:v1-27-130 60.7 KB > wrong helm app version which causing imagepullbackoff for package controler and refresher, see log_2
v1-27-129 other 17 days ago public.ecr.aws/eks-anywhere/...es-bundles:v1-27-129 60.7 KB > wrong helm app version which causing imagepullbackoff for package controler and refresher, see log_2
v1-27-128 other 2 months ago public.ecr.aws/eks-anywhere/...es-bundles:v1-27-128 > works fine
```

log_1
```
2023-11-02T10:52:24.780Z ECRCredInjector Failed to inject ECR credential to docker config {"error": "operation error ECR: GetAuthorizationToken, failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, static credentials are empty"}
github.com/aws/eks-anywhere-packages/pkg/registry.(*ECRCredInjector).Run
github.com/aws/eks-anywhere-packages/pkg/registry/ecr_cred_injector.go:56
```

log_2
```
eksa-packages eks-anywhere-packages 8 failed eks-anywhere-packages-0.0.0-8862036270224f2a6b8d6ecd455b6b1fa1084619 v0.0.0-8862036270224f2a6b8d6ecd455b6b1fa1084619
```
**What you expected to happen**:
eks-anywhere-packages shouldn't be published with such breaking changes
**How to reproduce it (as minimally and precisely as possible)**:
1. Using 0.17.4 eks-anywhere install k8s 1.27 using tinkerbell provider
2. Create eks-anywhere role along with anchor, follow https://anywhere.eks.amazonaws.com/docs/packages/credential-provider-package/iam_roles_anywhere/#prerequisites
3. Create aws-config secret in eks-packages ns
```
[default]
region = eu-west-1
credential_process = aws_signing_helper credential-process --certificate /var/lib/kubelet/pki/kubelet-client-current.pem --private-key /var/lib/kubelet/pki/kubelet-client-current.pem --profile-arn $PROFILE_ARN --role-arn $ROLE_ARN --trust-anchor-arn $TRUST_ANCHOR_ARN
```
4. Add package to download from private ECR registry
```
apiVersion: packages.eks.amazonaws.com/v1alpha1
kind: Package
metadata:
name: my-credential-provider-package
namespace: eksa-packages-eksa
annotations:
"helm.sh/resource-policy": keep
"anywhere.eks.aws.com/internal": "true"
spec:
packageName: credential-provider-package
targetNamespace: eksa-packages
config: |-
tolerations:
- key: "node-role.kubernetes.io/master"
operator: "Exists"
effect: "NoSchedule"
- key: "node-role.kubernetes.io/control-plane"
operator: "Exists"
effect: "NoSchedule"
sourceRegistry: public.ecr.aws/eks-anywhere
credential:
- matchImages:
- 000000000000.dkr.ecr.eu-west-2.amazonaws.com
profile: "default"
secretName: aws-config
defaultCacheDuration: "12h"
```
5. Verify you have latest version of packagebundle in packagebundlecontroller
6. Create pod with image from `000000000000.dkr.ecr.eu-west-2.amazonaws.com` registry
7. ImagePullBackOff should be logged from kubelet

**Anything else we need to know?**:
We also checked latest v1-28 with k8s 1.28 and it also experience issues the same as v1-27-137

**Environment**: k8s 1.27, tinker provisioner with bare bone nodes, ubuntu 22.04 ami
- EKS Anywhere Release: 0.17.4
- EKS Distro Release: -

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the credential-provider-package failures using the listed EKS Anywhere, Kubernetes, AWS configuration, and package bundle versions. Inspect the ECR credential path at pkg/registry/ecr_cred_injector.go:56 and the package bundle or Helm release flow mentioned in the report. Done means faulty bundles are not published and an end-to-end check detects these failures before release.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go, helm, kubernetes
Domain
cloud, devops, release, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.