bottlerocket-os / bottlerocket-os/bottlerocket
kubelet: Support running under a separate role.
- Dominant language
- Rust
- Stars
- 9.7k
- Forks
- 586
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 11
Description
**What I'd like:**
To be able to run the kubelet with a different role than the role attached to the node. Through a few small changes, we have been able to remove all permissions[1] directly attached to the instance role of our EKS nodes. I'll try to give an overview of the changes that we've done, and hopefully this is something that can be supported. I will note that we have this working with out hardened EKS 1.18 node (modifications are based off of https://github.com/aws-samples/amazon-eks-custom-amis), but it likely only makes sense to actually support this on 1.20+ given https://github.com/aws/eks-distro/issues/129. The following changes would likely be necessary:
1. A new configuration option for the kubelet role would be necessary.
1. An AWS config file with contents looking something like:
```toml
[profile kubelet]
credential_source = "Ec2InstanceMetadata"
role_arn = "${kubelet_role_arn}"
```
1. The following environment variables need to be set on the kubelet: `AWS_CONFIG_FILE` pointing at the config file above, and `AWS_PROFILE` set to the profile listed in the config file.
1. The environment for invoking `aws-iam-authenticator` needs to have the `AWS_PROFILE` set back to `default` so it will use the EC2 identity for authenticating with the cluster. This is needed to match the default settings applied to the kube-system/aws-auth ConfigMap.
It might be possible to do all of the assume-role configuration with a different setup, but this was what we were able to get to work.
With all of this in place, it should be possible to run the Bottlerocket VM with no permissions attached to the instance role, and instead have a second role that the kubelet uses for all of its needs. This extra layer of indirection helps to provide a little better permission isolation, and potentially some level of additional security[2].
We've had a hard time removing access to IMDS from pods (Datadog agent likes to give us trouble), so pulling the permissions to a different role prevents accidental permission use by pods in the cluster.
[1] Not exactly true given there is a role that the kubelet is assuming for operation, but this at least eliminates any easy to access permissions.
[2] I will concede that it may only be security through obscurity, but if my understanding is right it would require access to both the IMDS as well as discovering or knowing the role to assume.
**Any alternatives you've considered:**
Any other options for being able to run the node with zero permissions directly attached to the instance role would be welcome.
Contributor guide
Assessment
This issue has not been assessed yet.