aws / aws/aws-cdk

(aws-eks): `aws-auth` ConfigMap is still being replaced

Open
#19,218 7 comments 7 reactions 0 assignees View on GitHub
@aws-cdk/aws-eks effort/medium feature-request p1
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### What is the problem?

#7981 was not fixed by #8447 – The `aws-auth` ConfigMap is still being replaced, rather than appended to, any time the `Cluster.awsAuth` getter is invoked, which happens whenever you add a Fargate Profile to a cluster, forcing all Fargate Profiles to originate from the same instance of `Cluster` from `@aws-cdk/aws-eks/cluster.ts`.

In [the AwsAuth constructor](https://github.com/aws/aws-cdk/blob/04cf28ed7f9da1608d6d53c117d7b535bb1b299d/packages/%40aws-cdk/aws-eks/lib/aws-auth.ts#:~:text=*/-,export%20class%20AwsAuth%20extends%20Construct%20%7B,%7D,-/**), the following manifest is added added to the cluster:

```ts
new KubernetesResource(this, 'manifest', {
cluster: props.cluster,
manifest: [
{
apiVersion: 'v1',
kind: 'ConfigMap',
metadata: {
name: 'aws-auth',
namespace: 'kube-system',
},
data: {
mapRoles: this.synthesizeMapRoles(),
mapUsers: this.synthesizeMapUsers(),
mapAccounts: this.synthesizeMapAccounts(),
},
},
],
});
```

This constructor is invoked whenever the `Cluster.awsAuth` getter is invoked, which happens any time you invoke `Cluster.awsAuth.addRoleMapping`, which happens any time you invoke `Cluster.addFargateProfile`.

What this means is that if any software changes the `mapRoles` of the `aws-auth` ConfigMap without invoking `Cluster.awsAuth.addRoleMapping` on that particular instance of `Cluster`, those changes will be erased upon the next addition of a Fargate Profile.

In other words, upon each instantiation of `Cluster` where the `Cluster.awsAuth` getter is invoked, all ARNs of all `PodExecutionRole`s added manually with `kubectl`, other CDK apps, other IaC tools, an L1 Construct like `CfnFargateProfile`, etc. will be removed from the `aws-auth` ConfigMap, causing this error below, and preventing any new pods from spinning up.

> fargate-scheduler Misconfigured Fargate Profile: fargate profile blocked for new launches due to: Pod execution role is not found in auth config or does not have all required permissions for launching fargate pods.

### Reproduction Steps

See #7981

### What did you expect to happen?

See #7981

### What actually happened?

See #7981

### CDK CLI Version

1.85.0

### Framework Version

1.85.0

### Node.js Version

v14.17.1

### OS

macOS

### Language

Typescript

### Language Version

TypeScript (4.0.3)

### Other information

#### EKS version

1.19

#### Fix suggestion

Instead of applying a manifest to the cluster, an `AwsAuth` instance should create the `aws-auth` manifest if and only if the ConfigMap doesn't already exist in the cluster.

When `aws-auth` already exists, the result should be [a `kubectl patch` of the existing `aws-auth` ConfigMap](https://www.eksworkshop.com/intermediate/220_codepipeline/configmap/) (pictured below):

image

Contributor guide

Open the contributing guide

Research direction

Start by reading packages/@aws-cdk/aws-eks/lib/aws-auth.ts and cluster.ts, especially the AwsAuth constructor and the Cluster.awsAuth and addFargateProfile entry points. Review issue #7981 and its reproduction. Done means an existing aws-auth ConfigMap retains externally added roles while CDK additions are applied without replacing it.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, typescript
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.