bottlerocket-os / bottlerocket-os/bottlerocket

1.13.2: aws-iam-authenticator now finding `/root/.aws/config`

Open
#2,989 2 comments 0 reactions 0 assignees View on GitHub
area/core status/icebox type/bug
Dominant language
Rust
Stars
9.7k
Forks
586
Avg merge
1d 11h
Merged PRs (30d)
11

Description

**Image I'm using:**

Upgrading `Bottlerocket OS 1.13.1 (aws-k8s-1.22)` to `Bottlerocket OS 1.13.2 (aws-k8s-1.22)`

**What I expected to happen:**

A successful upgrade.

**What actually happened:**

Nodes failed to successfully authenticate with the cluster.

**How to reproduce the problem:**

I apologize that this might be long, and that the config/goals might feel a little odd.

Start nodes with the following configuration:

```toml
[settings.aws]
config = "W3Byb2ZpbGUgZGVmYXVsdF0KY3JlZGVudGlhbF9zb3VyY2UgPSBFYzJJbnN0YW5jZU1ldGFkYXRhCnJvbGVfYXJuID0gYXJuOmF3czppYW06OjEyMzQ1Njc4OTAxMjpyb2xlL015RXh0cmFSb2xlCg=="
profile = "default"
region = "us-east-1"
```

Where the `settings.aws.config` is something like:
```ini
[profile default]
credential_source = Ec2InstanceMetadata
role_arn = arn:aws:iam::123456789012:role/MyExtraRole
```

The role attached to the instance, and the additional role referenced above have all the correct permissions to allow the assume role to take place. Additionally, the instance role and the extra role have the same set of permissions attached to them (the required [AmazonEKSWorkerNodePolicy](https://us-east-1.console.aws.amazon.com/iam/home#/policies/arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy) and [AmazonEC2ContainerRegistryReadOnly](https://us-east-1.console.aws.amazon.com/iam/home#/policies/arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly)).

When the v1.13.1 instance starts up, the older v0.6.2 version of the `aws-iam-authenticator` would use the instance's role to generate the token needed to authenticate with the control plane. With the 1.13.2 version, which contains the v0.6.8 of the `aws-iam-authenticator`, it will discover the `/root/.aws/config` file and will make use of the default provider to create the token for authenticating the node with the control plane. This change in behavior was due to https://github.com/aws/aws-sdk-go/pull/4519 being included in the `aws-sdk-go` library upgrades. That PR adds a fallback to read the home directory information from `/etc/passwd` if the `HOME` environment variable is not set.

The assumed role is not able to authenticate with the control plane, even if the aws-auth ConfigMap is updated to support the different role, because the SessionName used by the assumed role does not contain the instance ID.

To get the instance to connect correctly, the AWS config file can be changed to use a profile with a name other than default. The use of `default` was suggested in https://github.com/bottlerocket-os/bottlerocket/issues/2885#issuecomment-1465226802. With https://github.com/bottlerocket-os/bottlerocket/pull/2904 and https://github.com/bottlerocket-os/bottlerocket/pull/2924 we were able to go back to using a name other than `default` which works.

I can see a few resolutions out of this issue, and mostly filing it to get feedback and consideration.
* Using a non-default profile is the best option. In other words, this isn't an issue, but just an odd interaction of components.
* Support getting the aws-iam-authenticator to set the session name to the right value so that the [`EC2PrivateDNSName`](https://github.com/kubernetes-sigs/aws-iam-authenticator/blob/77ef65a7c9ae850416e691998927451865e65b94/pkg/server/server.go#L433-L442) substitution works.
* Implement a more direct approach than the AWS config for achieving https://github.com/bottlerocket-os/bottlerocket/issues/1624.
* Some other option I'm not seeing.

It would be nice if we could get the `aws-iam-authenticator` to use the separated role for auth, but since that is just identity auth it isn't as immediately interesting as being able to remove all the policies attached to the instance role.

References:
* https://github.com/bottlerocket-os/bottlerocket/issues/2885 - prior attempts at setting `settings.aws.config`.
* https://github.com/bottlerocket-os/bottlerocket/issues/1624 - overall goal and why this sort of config is interesting.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.