aws-eks: Constructing a `AwsAuth` can completely break a cluster
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
We recently added a new `AwsAuth` construct to our cdk app. We were shocked to find that this broke all our node groups (as the new `AwsAuth` construct overwrote the aws-auth ConfigMap from cdk's [internally managed `AwsAuth` construct](https://github.com/aws/aws-cdk/blob/v2.114.1/packages/aws-cdk-lib/aws-eks/lib/cluster.ts#L1822)).
### Expected Behavior
I expect CDK to enforce that there is exactly one `AwsAuth` construct per cluster.
### Current Behavior
As mentioned above, the new `AwsAuth` construct clobbered the aws-auth ConfigMap managed by [the cluster's internal `AwsAuth` construct](https://github.com/aws/aws-cdk/blob/v2.114.1/packages/aws-cdk-lib/aws-eks/lib/cluster.ts#L1822).
### Reproduction Steps
Here's a simple cdk app that declares a cluster, a nodegroup for that cluster, and an explicit `AwsAuth` for that cluster:
```python
$ cat demo.py
import pprint
import aws_cdk as cdk
from aws_cdk import aws_ec2
from aws_cdk import aws_eks
from aws_cdk.assertions import Template
class DemoStack(cdk.Stack):
def __init__(self, scope):
super().__init__(scope, "DemoStack")
cluster = aws_eks.FargateCluster(
scope=self,
id="demo-cluster",
cluster_name="demo-cluster",
version=aws_eks.KubernetesVersion.V1_24,
)
cluster.add_nodegroup_capacity(
'demo-nodegroup',
nodegroup_name='demo-nodegroup',
instance_types=[aws_ec2.InstanceType('c6a.xlarge')],
)
aws_eks.AwsAuth(self, "problematic-aws-auth", cluster=cluster)
app = cdk.App()
demo_stack = DemoStack(app)
template = Template.from_stack(demo_stack)
custom_resources = template.find_resources("Custom::AWSCDK-EKS-KubernetesResource")
pprint.pprint(custom_resources)
```
Note how this produces 2 `Custom::AWSCDK-EKS-KubernetesResource` resources. These two resources will happily step on each other's toes (because of [this (unfortunately necessary) `overwrite: true`](https://github.com/aws/aws-cdk/blob/v2.114.1/packages/aws-cdk-lib/aws-eks/lib/aws-auth.ts#L38)).
```
$ python demo.py
{'democlusterAwsAuthmanifest59F406F8': {'DeletionPolicy': 'Delete',
'DependsOn': ['democlusterKubectlReadyBarrier409E0356'],
'Properties': {'ClusterName': {'Ref': 'democlusterE73BD733'},
'Manifest': {'Fn::Join': ['',
['[{"apiVersion":"v1","kind":"ConfigMap","metadata":{"name":"aws-auth","namespace":"kube-system","labels":{"aws.cdk.eks/prune-c8e20f52ee1502bfdc03c85de68fda3a9cbb05b58d":""}},"data":{"mapRoles":"[{\\"rolearn\\":\\"',
{'Fn::GetAtt': ['democlusterfargateprofiledefaultPodExecutionRoleB482677B',
'Arn']},
'\\",\\"username\\":\\"system:node:{{SessionName}}\\",\\"groups\\":[\\"system:bootstrappers\\",\\"system:nodes\\",\\"system:node-proxier\\"]},{\\"rolearn\\":\\"',
{'Fn::GetAtt': ['democlusterNodegroupdemonodegroupNodeGroupRole6A5CDA1C',
'Arn']},
'\\",\\"username\\":\\"system:node:{{EC2PrivateDNSName}}\\",\\"groups\\":[\\"system:bootstrappers\\",\\"system:nodes\\"]}]","mapUsers":"[]","mapAccounts":"[]"}}]']]},
'Overwrite': True,
'PruneLabel': 'aws.cdk.eks/prune-c8e20f52ee1502bfdc03c85de68fda3a9cbb05b58d',
'RoleArn': {'Fn::GetAtt': ['democlusterCreationRole733E3675',
'Arn']},
'ServiceToken': {'Fn::GetAtt': ['awscdkawseksKubectlProviderNestedStackawscdkawseksKubectlProviderNestedStackResourceA7AEBA6B',
'Outputs.DemoStackawscdkawseksKubectlProviderframeworkonEvent705EBCF2Arn']}},
'Type': 'Custom::AWSCDK-EKS-KubernetesResource',
'UpdateReplacePolicy': 'Delete'},
'problematicawsauthmanifestEB215835': {'DeletionPolicy': 'Delete',
'DependsOn': ['democlusterKubectlReadyBarrier409E0356'],
'Properties': {'ClusterName': {'Ref': 'democlusterE73BD733'},
'Manifest': '[{"apiVersion":"v1","kind":"ConfigMap","metadata":{"name":"aws-auth","namespace":"kube-system","labels":{"aws.cdk.eks/prune-c8993b9de23d4ff03256c14b261e46b123d26fe727":""}},"data":{"mapRoles":"[]","mapUsers":"[]","mapAccounts":"[]"}}]',
'Overwrite': True,
'PruneLabel': 'aws.cdk.eks/prune-c8993b9de23d4ff03256c14b261e46b123d26fe727',
'RoleArn': {'Fn::GetAtt': ['democlusterCreationRole733E3675',
'Arn']},
'ServiceToken': {'Fn::GetAtt': ['awscdkawseksKubectlProviderNestedStackawscdkawseksKubectlProviderNestedStackResourceA7AEBA6B',
'Outputs.DemoStackawscdkawseksKubectlProviderframeworkonEvent705EBCF2Arn']}},
'Type': 'Custom::AWSCDK-EKS-KubernetesResource',
'UpdateReplacePolicy': 'Delete'}}
```
### Possible Solution
I have read through https://github.com/aws/aws-cdk/issues/19218, which talks about a similar issue (the aws-auth ConfigMap getting unexpectedly clobbered), but that's all about using multiple tools to manage a cluster. In the scenario described here, we're using exactly 1 CDK app to manage our cluster, and IMO, CDK should protect us against this.
Some ideas for how to fix this:
1. When constructing a `AwsAuth` for a cluster, check if there's *already* a `AwsAuth` instantiated for that cluster. If there is one already, crash.
2. Don't expose the `AwsAuth` construct at all, just keep it as an internal implementation detail of Cluster. As I understand things, it's never something you'd want to use directly when you've declared your cluster in cdk (instead you'd want to interact with the underlying `cluster.awsAuth` attribute [as documented here](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_eks-readme.html#aws-iam-mapping)).
3. (what we're going to do until this is addressed in cdk itself) Add a post-synthesis step that loops over all `AwsAuth` constructs in the tree and asserts that they're for different clusters.
### Additional Information/Context
_No response_
### CDK CLI Version
2.109.0 (build 941dc16)
### Framework Version
_No response_
### Node.js Version
v18.15.0
### OS
N/A (this happens both on macOS and Linux)
### Language
Python
### Language Version
Python (3.10.6)
### Other information
_No response_
Contributor guide
Research direction
Start with packages/aws-cdk-lib/aws-eks/lib/cluster.ts and aws-auth.ts, then run the Python reproduction to inspect the two generated Custom::AWSCDK-EKS-KubernetesResource resources. Review existing EKS tests for cluster authentication behavior. Done means a cluster cannot be left with competing AwsAuth resources and the regression is covered by a test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, typescript
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100