aws-eks: KubernetesManifest Overwrite option invalid now that ServerSideApply is defaulted
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
At some point recently, the `kubectl` CLI started setting `--server-side-apply` as the default behavior. The problem is that with https://github.com/kubernetes/kubernetes/issues/44165, `kubectl apply -f ...` no longer works the way you'd expect. On a server-side-apply, it seems that Kubernetes will refuse to update a resource that already exists, which then reports back to the Lambda function an `AlreadyExists` error.
### Regression Issue
- [X] Select this option if this issue appears to be a regression.
### Last Known Working CDK Version
unknown
### Expected Behavior
I would expect that `kubectl apply -f ...` just works ... (which is configured by setting `overwrite: true` on the `KubernetesManifest` resource)... but instead it's failing.
### Current Behavior
Here are the logs from the Lambda function trying to run `kubectl apply -f` on a resource that happens to already exist in the cluster:
```
[INFO] 2024-10-05T18:12:44.871Z bc961513-6915-4749-a32f-a787912469b1 Running command: ['kubectl', 'apply', '--kubeconfig', '/tmp/kubeconfig', '-f', '/tmp/manifest.yaml']
[INFO] 2024-10-05T18:12:44.871Z bc961513-6915-4749-a32f-a787912469b1 manifest written to: /tmp/manifest.yaml
[INFO] 2024-10-05T18:12:42.741Z bc961513-6915-4749-a32f-a787912469b1 Running command: ['aws', 'eks', 'update-kubeconfig', '--role-arn', 'arn:aws:iam::...:role/...-3bVbyuZfXXf4', '--name', '....', '--kubeconfig', '/tmp/kubeconfig']
[INFO] 2024-10-05T18:12:42.740Z bc961513-6915-4749-a32f-a787912469b1 {"RequestType": "Create", "ServiceToken": "arn:aws:lambda:us-west-2:...:function:INFRA...-oNQV5X2TDIjj", "ResponseURL": "...", "StackId": "arn:aws:cloudformation:us-west-2:...:stack/...-ContinuousDeploymentNestedStackContinuousDeploymentNes-8QONVDS1QSK3/4fb3cbe0-8345-11ef-afa7-067d0aea149f", "RequestId": "ec9880d5-f9bf-498d-9780-12134c81f17d", "LogicalResourceId": "ArgoCDSystemPostHelmResources073D6BA8", "ResourceType": "Custom::AWSCDK-EKS-KubernetesResource", "ResourceProperties": {"ServiceToken": "arn:aws:lambda:us-west-2:...:function:...-oNQV5X2TDIjj", "Overwrite": "true", "PruneLabel": "aws.cdk.eks/prune-c8c67f619695e93fb41d90faa4dabab90eb2bca3ea", "ClusterName": "...", "Manifest": "[{\"apiVersion\":\"argoproj.io/v1alpha1\",\"kind\":\"AppProject\",\"metadata\":{\"name\":\"default\",\"namespace\":\"argocd-system\",\"labels\":{\"aws.cdk.eks/prune-c8c67f619695e93fb41d90faa4dabab90eb2bca3ea\":\"\"}},\"spec\":{\"clusterResourceWhitelist\":[{\"group\":\"*\",\"kind\":\"*\"}],\"destinations\":[{\"namespace\":\"*\",\"server\":\"https://kubernetes.default.svc\"}],\"sourceRepos\":[\"*\"]}}]", "RoleArn": "arn:aws:iam::...:role/...-3bVbyuZfXXf4"}}
{
"RequestType": "Create",
"ServiceToken": "arn:aws:lambda:us-west-2:...:function:...-oNQV5X2TDIjj",
"ResponseURL": "...",
"StackId": "arn:aws:cloudformation:us-west-2:...:stack/...-ContinuousDeploymentNestedStackContinuousDeploymentNes-8QONVDS1QSK3/4fb3cbe0-8345-11ef-afa7-067d0aea149f",
"RequestId": "ec9880d5-f9bf-498d-9780-12134c81f17d",
"LogicalResourceId": "ArgoCDSystemPostHelmResources073D6BA8",
"ResourceType": "Custom::AWSCDK-EKS-KubernetesResource",
"ResourceProperties": {
"ServiceToken": "arn:aws:lambda:us-west-2:...:function:...-oNQV5X2TDIjj",
"Overwrite": "true",
"PruneLabel": "aws.cdk.eks/prune-c8c67f619695e93fb41d90faa4dabab90eb2bca3ea",
"ClusterName": "...",
"Manifest": "[{\"apiVersion\":\"argoproj.io/v1alpha1\",\"kind\":\"AppProject\",\"metadata\":{\"name\":\"default\",\"namespace\":\"argocd-system\",\"labels\":{\"aws.cdk.eks/prune-c8c67f619695e93fb41d90faa4dabab90eb2bca3ea\":\"\"}},\"spec\":{\"clusterResourceWhitelist\":[{\"group\":\"*\",\"kind\":\"*\"}],\"destinations\":[{\"namespace\":\"*\",\"server\":\"https://kubernetes.default.svc\"}],\"sourceRepos\":[\"*\"]}}]",
"RoleArn": "arn:aws:iam::...:role/...-3bVbyuZfXXf4"
}
}
[ERROR] Exception: b'Error from server (AlreadyExists): error when creating "/tmp/manifest.yaml": appprojects.argoproj.io "default" already exists\n'
Traceback (most recent call last):
File "/var/task/index.py", line 14, in handler
return apply_handler(event, context)
File "/var/task/apply/__init__.py", line 60, in apply_handler
kubectl('apply', manifest_file, *kubectl_opts)
File "/var/task/apply/__init__.py", line 91, in kubectl
raise Exception(output)
```
If we go and look at the Kubernetes Audit logs, we can see that ArgoCD first creates this `default` resource, and then the next `Create` call fails with a `409` and is a server-side-apply call:
Here's the first `create` call (made by ArgoCD, and uncontrollable by us)
```json
{
"kind": "Event",
"apiVersion": "audit.k8s.io/v1",
"level": "Metadata",
"auditID": "21c5b6a7-eeee-4676-b7e7-2764d16517f1",
"stage": "ResponseComplete",
"requestURI": "/apis/argoproj.io/v1alpha1/namespaces/argocd-system/appprojects",
"verb": "create",
"user": {
"username": "system:serviceaccount:argocd-system:argocd-server",
"uid": "ec844b9d-8ce1-4325-905a-479df42f0aed",
"groups": [
"system:serviceaccounts",
"system:serviceaccounts:argocd-system",
"system:authenticated"
],
"extra": {
...
}
},
"sourceIPs": [
"..."
],
"userAgent": "argocd-server/v0.0.0 (linux/arm64) kubernetes/$Format",
"objectRef": {
"resource": "appprojects",
"namespace": "argocd-system",
"name": "default",
"apiGroup": "argoproj.io",
"apiVersion": "v1alpha1"
},
"responseStatus": {
"metadata": {},
"code": 201
},
"requestReceivedTimestamp": "2024-10-05T18:12:49.590120Z",
"stageTimestamp": "2024-10-05T18:12:49.594167Z",
"annotations": {
"authorization.k8s.io/decision": "allow",
"authorization.k8s.io/reason": "RBAC: allowed by RoleBinding \"argocd-system-server/argocd-system\" of Role \"argocd-system-server\" to ServiceAccount \"argocd-server/argocd-system\""
}
}
```
Then we see a second call, this time via `kubectl`... note the `kubectl-client-side-apply` in the `requestURI` path:
```json
{
"kind": "Event",
"apiVersion": "audit.k8s.io/v1",
"level": "Metadata",
"auditID": "64c914a4-e184-4079-a6b6-8c5630e473fd",
"stage": "ResponseComplete",
"requestURI": "/apis/argoproj.io/v1alpha1/namespaces/argocd-system/appprojects?fieldManager=kubectl-client-side-apply&fieldValidation=Strict",
"verb": "create",
"user": {
"username": "arn:aws:sts::...:assumed-role/.../EKSGetTokenAuth",
"uid": "aws-iam-authenticator:...:...",
"groups": [
"system:authenticated"
],
"extra": {
...
}
},
"sourceIPs": [
"..."
],
"userAgent": "kubectl/v1.28.3 (linux/amd64) kubernetes/a8a1abc",
"objectRef": {
"resource": "appprojects",
"namespace": "argocd-system",
"name": "default",
"apiGroup": "argoproj.io",
"apiVersion": "v1alpha1"
},
"responseStatus": {
"metadata": {},
"status": "Failure",
"message": "appprojects.argoproj.io \"default\" already exists",
"reason": "AlreadyExists",
"details": {
"name": "default",
"group": "argoproj.io",
"kind": "appprojects"
},
"code": 409
},
"requestReceivedTimestamp": "2024-10-05T18:12:49.605439Z",
"stageTimestamp": "2024-10-05T18:12:49.614296Z",
"annotations": {
"authorization.k8s.io/decision": "allow",
"authorization.k8s.io/reason": "EKS Access Policy: allowed by ClusterRoleBinding \"arn:aws:iam::...:role/...+arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy\" of ClusterRole \"arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy\" to User \"...\""
}
}
```
### Reproduction Steps
N/A
### Possible Solution
I think that when `overwrite: true` is set, then the `--server-side=false` flag should also be applied to the command..
### Additional Information/Context
_No response_
### CDK CLI Version
2.161.1
### Framework Version
_No response_
### Node.js Version
18
### OS
linux
### Language
TypeScript
### Language Version
_No response_
### Other information
_No response_
Contributor guide
Research direction
Start with the Lambda apply handler shown in apply/__init__.py, especially apply_handler and the kubectl('apply', manifest_file, *kubectl_opts) call, then trace how KubernetesManifest's overwrite setting becomes kubectl options. Check the flag behavior described in the logs and existing coverage for this path; done means overwrite=true updates an existing resource without an AlreadyExists error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes, python, typescript
- Domain
- cloud, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100