aws / aws/aws-cdk

aws-eks: Constructing a `AwsAuth` can completely break a cluster

Open
#28,333 8 comments 2 reactions 0 assignees View on GitHub
@aws-cdk/aws-eks bug p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

We recently added a new `AwsAuth` construct to our cdk app. We were shocked to find that this broke all our node groups (as the new `AwsAuth` construct overwrote the aws-auth ConfigMap from cdk's [internally managed `AwsAuth` construct](https://github.com/aws/aws-cdk/blob/v2.114.1/packages/aws-cdk-lib/aws-eks/lib/cluster.ts#L1822)).

### Expected Behavior

I expect CDK to enforce that there is exactly one `AwsAuth` construct per cluster.

### Current Behavior

As mentioned above, the new `AwsAuth` construct clobbered the aws-auth ConfigMap managed by [the cluster's internal `AwsAuth` construct](https://github.com/aws/aws-cdk/blob/v2.114.1/packages/aws-cdk-lib/aws-eks/lib/cluster.ts#L1822).

### Reproduction Steps

Here's a simple cdk app that declares a cluster, a nodegroup for that cluster, and an explicit `AwsAuth` for that cluster:

```python
$ cat demo.py
import pprint

import aws_cdk as cdk
from aws_cdk import aws_ec2
from aws_cdk import aws_eks
from aws_cdk.assertions import Template

class DemoStack(cdk.Stack):
def __init__(self, scope):
super().__init__(scope, "DemoStack")

cluster = aws_eks.FargateCluster(
scope=self,
id="demo-cluster",
cluster_name="demo-cluster",
version=aws_eks.KubernetesVersion.V1_24,
)

cluster.add_nodegroup_capacity(
'demo-nodegroup',
nodegroup_name='demo-nodegroup',
instance_types=[aws_ec2.InstanceType('c6a.xlarge')],
)

aws_eks.AwsAuth(self, "problematic-aws-auth", cluster=cluster)

app = cdk.App()
demo_stack = DemoStack(app)

template = Template.from_stack(demo_stack)
custom_resources = template.find_resources("Custom::AWSCDK-EKS-KubernetesResource")
pprint.pprint(custom_resources)
```

Note how this produces 2 `Custom::AWSCDK-EKS-KubernetesResource` resources. These two resources will happily step on each other's toes (because of [this (unfortunately necessary) `overwrite: true`](https://github.com/aws/aws-cdk/blob/v2.114.1/packages/aws-cdk-lib/aws-eks/lib/aws-auth.ts#L38)).

```
$ python demo.py
{'democlusterAwsAuthmanifest59F406F8': {'DeletionPolicy': 'Delete',
'DependsOn': ['democlusterKubectlReadyBarrier409E0356'],
'Properties': {'ClusterName': {'Ref': 'democlusterE73BD733'},
'Manifest': {'Fn::Join': ['',
['[{"apiVersion":"v1","kind":"ConfigMap","metadata":{"name":"aws-auth","namespace":"kube-system","labels":{"aws.cdk.eks/prune-c8e20f52ee1502bfdc03c85de68fda3a9cbb05b58d":""}},"data":{"mapRoles":"[{\\"rolearn\\":\\"',
{'Fn::GetAtt': ['democlusterfargateprofiledefaultPodExecutionRoleB482677B',
'Arn']},
'\\",\\"username\\":\\"system:node:{{SessionName}}\\",\\"groups\\":[\\"system:bootstrappers\\",\\"system:nodes\\",\\"system:node-proxier\\"]},{\\"rolearn\\":\\"',
{'Fn::GetAtt': ['democlusterNodegroupdemonodegroupNodeGroupRole6A5CDA1C',
'Arn']},
'\\",\\"username\\":\\"system:node:{{EC2PrivateDNSName}}\\",\\"groups\\":[\\"system:bootstrappers\\",\\"system:nodes\\"]}]","mapUsers":"[]","mapAccounts":"[]"}}]']]},
'Overwrite': True,
'PruneLabel': 'aws.cdk.eks/prune-c8e20f52ee1502bfdc03c85de68fda3a9cbb05b58d',
'RoleArn': {'Fn::GetAtt': ['democlusterCreationRole733E3675',
'Arn']},
'ServiceToken': {'Fn::GetAtt': ['awscdkawseksKubectlProviderNestedStackawscdkawseksKubectlProviderNestedStackResourceA7AEBA6B',
'Outputs.DemoStackawscdkawseksKubectlProviderframeworkonEvent705EBCF2Arn']}},
'Type': 'Custom::AWSCDK-EKS-KubernetesResource',
'UpdateReplacePolicy': 'Delete'},
'problematicawsauthmanifestEB215835': {'DeletionPolicy': 'Delete',
'DependsOn': ['democlusterKubectlReadyBarrier409E0356'],
'Properties': {'ClusterName': {'Ref': 'democlusterE73BD733'},
'Manifest': '[{"apiVersion":"v1","kind":"ConfigMap","metadata":{"name":"aws-auth","namespace":"kube-system","labels":{"aws.cdk.eks/prune-c8993b9de23d4ff03256c14b261e46b123d26fe727":""}},"data":{"mapRoles":"[]","mapUsers":"[]","mapAccounts":"[]"}}]',
'Overwrite': True,
'PruneLabel': 'aws.cdk.eks/prune-c8993b9de23d4ff03256c14b261e46b123d26fe727',
'RoleArn': {'Fn::GetAtt': ['democlusterCreationRole733E3675',
'Arn']},
'ServiceToken': {'Fn::GetAtt': ['awscdkawseksKubectlProviderNestedStackawscdkawseksKubectlProviderNestedStackResourceA7AEBA6B',
'Outputs.DemoStackawscdkawseksKubectlProviderframeworkonEvent705EBCF2Arn']}},
'Type': 'Custom::AWSCDK-EKS-KubernetesResource',
'UpdateReplacePolicy': 'Delete'}}
```

### Possible Solution

I have read through https://github.com/aws/aws-cdk/issues/19218, which talks about a similar issue (the aws-auth ConfigMap getting unexpectedly clobbered), but that's all about using multiple tools to manage a cluster. In the scenario described here, we're using exactly 1 CDK app to manage our cluster, and IMO, CDK should protect us against this.

Some ideas for how to fix this:

1. When constructing a `AwsAuth` for a cluster, check if there's *already* a `AwsAuth` instantiated for that cluster. If there is one already, crash.
2. Don't expose the `AwsAuth` construct at all, just keep it as an internal implementation detail of Cluster. As I understand things, it's never something you'd want to use directly when you've declared your cluster in cdk (instead you'd want to interact with the underlying `cluster.awsAuth` attribute [as documented here](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_eks-readme.html#aws-iam-mapping)).
3. (what we're going to do until this is addressed in cdk itself) Add a post-synthesis step that loops over all `AwsAuth` constructs in the tree and asserts that they're for different clusters.

### Additional Information/Context

_No response_

### CDK CLI Version

2.109.0 (build 941dc16)

### Framework Version

_No response_

### Node.js Version

v18.15.0

### OS

N/A (this happens both on macOS and Linux)

### Language

Python

### Language Version

Python (3.10.6)

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with packages/aws-cdk-lib/aws-eks/lib/cluster.ts and aws-auth.ts, then run the Python reproduction to inspect the two generated Custom::AWSCDK-EKS-KubernetesResource resources. Review existing EKS tests for cluster authentication behavior. Done means a cluster cannot be left with competing AwsAuth resources and the regression is covered by a test.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.