aws / aws/aws-cdk

aws-eks: KubernetesManifest Overwrite option invalid now that ServerSideApply is defaulted

Open
#31,697 6 comments 1 reaction 0 assignees View on GitHub
@aws-cdk/aws-eks @aws-cdk/aws-eks-v2-alpha bug p3 potential-regression
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

At some point recently, the `kubectl` CLI started setting `--server-side-apply` as the default behavior. The problem is that with https://github.com/kubernetes/kubernetes/issues/44165, `kubectl apply -f ...` no longer works the way you'd expect. On a server-side-apply, it seems that Kubernetes will refuse to update a resource that already exists, which then reports back to the Lambda function an `AlreadyExists` error.

### Regression Issue

- [X] Select this option if this issue appears to be a regression.

### Last Known Working CDK Version

unknown

### Expected Behavior

I would expect that `kubectl apply -f ...` just works ... (which is configured by setting `overwrite: true` on the `KubernetesManifest` resource)... but instead it's failing.

### Current Behavior

Here are the logs from the Lambda function trying to run `kubectl apply -f` on a resource that happens to already exist in the cluster:

```
[INFO] 2024-10-05T18:12:44.871Z bc961513-6915-4749-a32f-a787912469b1 Running command: ['kubectl', 'apply', '--kubeconfig', '/tmp/kubeconfig', '-f', '/tmp/manifest.yaml']
[INFO] 2024-10-05T18:12:44.871Z bc961513-6915-4749-a32f-a787912469b1 manifest written to: /tmp/manifest.yaml
[INFO] 2024-10-05T18:12:42.741Z bc961513-6915-4749-a32f-a787912469b1 Running command: ['aws', 'eks', 'update-kubeconfig', '--role-arn', 'arn:aws:iam::...:role/...-3bVbyuZfXXf4', '--name', '....', '--kubeconfig', '/tmp/kubeconfig']
[INFO] 2024-10-05T18:12:42.740Z bc961513-6915-4749-a32f-a787912469b1 {"RequestType": "Create", "ServiceToken": "arn:aws:lambda:us-west-2:...:function:INFRA...-oNQV5X2TDIjj", "ResponseURL": "...", "StackId": "arn:aws:cloudformation:us-west-2:...:stack/...-ContinuousDeploymentNestedStackContinuousDeploymentNes-8QONVDS1QSK3/4fb3cbe0-8345-11ef-afa7-067d0aea149f", "RequestId": "ec9880d5-f9bf-498d-9780-12134c81f17d", "LogicalResourceId": "ArgoCDSystemPostHelmResources073D6BA8", "ResourceType": "Custom::AWSCDK-EKS-KubernetesResource", "ResourceProperties": {"ServiceToken": "arn:aws:lambda:us-west-2:...:function:...-oNQV5X2TDIjj", "Overwrite": "true", "PruneLabel": "aws.cdk.eks/prune-c8c67f619695e93fb41d90faa4dabab90eb2bca3ea", "ClusterName": "...", "Manifest": "[{\"apiVersion\":\"argoproj.io/v1alpha1\",\"kind\":\"AppProject\",\"metadata\":{\"name\":\"default\",\"namespace\":\"argocd-system\",\"labels\":{\"aws.cdk.eks/prune-c8c67f619695e93fb41d90faa4dabab90eb2bca3ea\":\"\"}},\"spec\":{\"clusterResourceWhitelist\":[{\"group\":\"*\",\"kind\":\"*\"}],\"destinations\":[{\"namespace\":\"*\",\"server\":\"https://kubernetes.default.svc\"}],\"sourceRepos\":[\"*\"]}}]", "RoleArn": "arn:aws:iam::...:role/...-3bVbyuZfXXf4"}}
{
"RequestType": "Create",
"ServiceToken": "arn:aws:lambda:us-west-2:...:function:...-oNQV5X2TDIjj",
"ResponseURL": "...",
"StackId": "arn:aws:cloudformation:us-west-2:...:stack/...-ContinuousDeploymentNestedStackContinuousDeploymentNes-8QONVDS1QSK3/4fb3cbe0-8345-11ef-afa7-067d0aea149f",
"RequestId": "ec9880d5-f9bf-498d-9780-12134c81f17d",
"LogicalResourceId": "ArgoCDSystemPostHelmResources073D6BA8",
"ResourceType": "Custom::AWSCDK-EKS-KubernetesResource",
"ResourceProperties": {
"ServiceToken": "arn:aws:lambda:us-west-2:...:function:...-oNQV5X2TDIjj",
"Overwrite": "true",
"PruneLabel": "aws.cdk.eks/prune-c8c67f619695e93fb41d90faa4dabab90eb2bca3ea",
"ClusterName": "...",
"Manifest": "[{\"apiVersion\":\"argoproj.io/v1alpha1\",\"kind\":\"AppProject\",\"metadata\":{\"name\":\"default\",\"namespace\":\"argocd-system\",\"labels\":{\"aws.cdk.eks/prune-c8c67f619695e93fb41d90faa4dabab90eb2bca3ea\":\"\"}},\"spec\":{\"clusterResourceWhitelist\":[{\"group\":\"*\",\"kind\":\"*\"}],\"destinations\":[{\"namespace\":\"*\",\"server\":\"https://kubernetes.default.svc\"}],\"sourceRepos\":[\"*\"]}}]",
"RoleArn": "arn:aws:iam::...:role/...-3bVbyuZfXXf4"
}
}
[ERROR] Exception: b'Error from server (AlreadyExists): error when creating "/tmp/manifest.yaml": appprojects.argoproj.io "default" already exists\n'
Traceback (most recent call last):
  File "/var/task/index.py", line 14, in handler
    return apply_handler(event, context)
  File "/var/task/apply/__init__.py", line 60, in apply_handler
    kubectl('apply', manifest_file, *kubectl_opts)
  File "/var/task/apply/__init__.py", line 91, in kubectl
    raise Exception(output)
```

If we go and look at the Kubernetes Audit logs, we can see that ArgoCD first creates this `default` resource, and then the next `Create` call fails with a `409` and is a server-side-apply call:

Here's the first `create` call (made by ArgoCD, and uncontrollable by us)
```json
{
"kind": "Event",
"apiVersion": "audit.k8s.io/v1",
"level": "Metadata",
"auditID": "21c5b6a7-eeee-4676-b7e7-2764d16517f1",
"stage": "ResponseComplete",
"requestURI": "/apis/argoproj.io/v1alpha1/namespaces/argocd-system/appprojects",
"verb": "create",
"user": {
"username": "system:serviceaccount:argocd-system:argocd-server",
"uid": "ec844b9d-8ce1-4325-905a-479df42f0aed",
"groups": [
"system:serviceaccounts",
"system:serviceaccounts:argocd-system",
"system:authenticated"
],
"extra": {
...
}
},
"sourceIPs": [
"..."
],
"userAgent": "argocd-server/v0.0.0 (linux/arm64) kubernetes/$Format",
"objectRef": {
"resource": "appprojects",
"namespace": "argocd-system",
"name": "default",
"apiGroup": "argoproj.io",
"apiVersion": "v1alpha1"
},
"responseStatus": {
"metadata": {},
"code": 201
},
"requestReceivedTimestamp": "2024-10-05T18:12:49.590120Z",
"stageTimestamp": "2024-10-05T18:12:49.594167Z",
"annotations": {
"authorization.k8s.io/decision": "allow",
"authorization.k8s.io/reason": "RBAC: allowed by RoleBinding \"argocd-system-server/argocd-system\" of Role \"argocd-system-server\" to ServiceAccount \"argocd-server/argocd-system\""
}
}
```

Then we see a second call, this time via `kubectl`... note the `kubectl-client-side-apply` in the `requestURI` path:

```json
{
"kind": "Event",
"apiVersion": "audit.k8s.io/v1",
"level": "Metadata",
"auditID": "64c914a4-e184-4079-a6b6-8c5630e473fd",
"stage": "ResponseComplete",
"requestURI": "/apis/argoproj.io/v1alpha1/namespaces/argocd-system/appprojects?fieldManager=kubectl-client-side-apply&fieldValidation=Strict",
"verb": "create",
"user": {
"username": "arn:aws:sts::...:assumed-role/.../EKSGetTokenAuth",
"uid": "aws-iam-authenticator:...:...",
"groups": [
"system:authenticated"
],
"extra": {
...
}
},
"sourceIPs": [
"..."
],
"userAgent": "kubectl/v1.28.3 (linux/amd64) kubernetes/a8a1abc",
"objectRef": {
"resource": "appprojects",
"namespace": "argocd-system",
"name": "default",
"apiGroup": "argoproj.io",
"apiVersion": "v1alpha1"
},
"responseStatus": {
"metadata": {},
"status": "Failure",
"message": "appprojects.argoproj.io \"default\" already exists",
"reason": "AlreadyExists",
"details": {
"name": "default",
"group": "argoproj.io",
"kind": "appprojects"
},
"code": 409
},
"requestReceivedTimestamp": "2024-10-05T18:12:49.605439Z",
"stageTimestamp": "2024-10-05T18:12:49.614296Z",
"annotations": {
"authorization.k8s.io/decision": "allow",
"authorization.k8s.io/reason": "EKS Access Policy: allowed by ClusterRoleBinding \"arn:aws:iam::...:role/...+arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy\" of ClusterRole \"arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy\" to User \"...\""
}
}
```

### Reproduction Steps

N/A

### Possible Solution

I think that when `overwrite: true` is set, then the `--server-side=false` flag should also be applied to the command..

### Additional Information/Context

_No response_

### CDK CLI Version

2.161.1

### Framework Version

_No response_

### Node.js Version

18

### OS

linux

### Language

TypeScript

### Language Version

_No response_

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the Lambda apply handler shown in apply/__init__.py, especially apply_handler and the kubectl('apply', manifest_file, *kubectl_opts) call, then trace how KubernetesManifest's overwrite setting becomes kubectl options. Check the flag behavior described in the logs and existing coverage for this path; done means overwrite=true updates an existing resource without an AlreadyExists error.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, python, typescript
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.