[EKS] [request]: Support worker groups in separate AWS account
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
**Tell us about your request**
EKS (likely via the AWS cloud controller) should support worker nodes that live in another AWS account and join a cluster either via a shared or peered VPC.
Currently, this isn't possible, as the cloud controller won't be able to describe the instance, add ENIs or EBS volumes, and probably other stuff.
This is somewhat related to (or could perhaps also solve) https://github.com/aws/containers-roadmap/issues/153
This is likely best done by having some label/annotation on nodes (and probably other resources like PersistentVolumes) that would indicate to the cloud controller that it should assume a role to be able interact with the underlying cloud resource.
**Which service(s) is this request for?**
EKS, but hopefully it would be part of the AWS could controller
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
What outcome are you trying to achieve, ultimately, and why is it hard/impossible to do right now? What is the impact of not having this problem solved? The more details you can provide, the better we'll be able to understand and solve the problem.
For the last few years, AWS has been encouraging users to move to more AWS accounts as a way to help organizations make permissions easier and isolate apps. AWS has also provided new features (such as VPC sharing and organizations) that making having lots of accounts easier.
At the same time, EKS/K8S adoption is happening, which provides it's own powerful mechanism for isolation of compute into namespaces, but for an organization that is also moving toward more granular AWS accounts, these two trends are hard to rectify.
Essentially, we would prefer to give teams (or even individual services) their own individual AWS account, which would allow teams to have more control and manage resources they need easily. At the same time, we want to join that compute to a cluster and mange it (as well as PV EBS, LB Server integrations, etc) natively within K8S, which gives us broader organizational visibility as well as the ability to connect services together. While lots of clusters tied together with federation is perhaps an option, it seems like there should be more obvious path for this use case.
**Are you currently working around this issue?**
How are you currently solving this problem?
We aren't, but it is one question we have started thinking about.
We have thought it would be feasible to use a shared account that would host EKS/K8S clusters. These accounts would also own all the compute and we will figure out the best way to give teams access into this account that gives them the ability to manage compute pools without impacting other teams (i.e. they can create and destroy their own ASGs, but not others).
For all other AWS resources, we could then create more granular AWS accounts with a shared or peered VPC and use something like kiam (or iam roles per pod once available) to allow pods to assume a role that would give it access to other AWS resources.
We think this is workable, but it has some complexity and requires a fair amount of work on our side.
**Additional context**
Having spent some time looking at the aws cloud controller, it seems like this could be done through labels/annotations added to node and other resources that would indicate a role for the cloud controller to assume whenever it had to interact with that resource.
Contributor guide
Research direction
Start by reviewing the AWS cloud controller and the related issue 153. Investigate how nodes, PersistentVolumes, ENIs, EBS volumes, and load balancers are handled across AWS accounts, and determine the required scope for role assumption. Done would require an agreed design and implementation path for cross-account EKS resource management.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100