[EKS] [Fargate] [request]: Make Fargate look like a normal k8s Node/Nodes
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
I'd like Fargate to appear as a Node or Nodes in my EKS cluster, and hence interact natively with things like `nodeSelector`, `taints` and `tolerations`, `podAffinity`, [metrics](https://github.com/aws/containers-roadmap/issues/942), [Pod Topology Spread Constraints](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/), etc.
i.e. the closer I can come to having Fargate feel like a special NodeGroup (or really, a Managed NodeGroup, once the various missing features there are ironed out), the more natural and easy it is to use in a cluster.
As an exempler, if a Fargate Profile (without the selector) was exposed as a tainted and labelled Node, then allowing Pods to run on Kubelet would only require the same tolerations as used to allow a Pod to use any specialised Nodes in Kubernetes, and forcing a Pod to run on Fargate would only require the same nodeSelector as used to lock a Pod to any specialised Nodes in Kubernetes; i.e. the [existing Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) "just works".
It seems like the [virtual-kubelet](https://github.com/virtual-kubelet/virtual-kubelet) project is intended to offer [exactly that](https://github.com/virtual-kubelet/aws-fargate/blob/master/fargate.toml), but the [AWS Fargate provider](https://github.com/virtual-kubelet/aws-fargate) is abandoned, despite an interesting [blog post demonstrating it with a non-EKS AWS environment in 2018](https://aws.amazon.com/blogs/opensource/aws-fargate-virtual-kubelet/).
A brief poke at Virtual Kubelet suggests that the provider-specific implementation would need an [Admission Controller](https://github.com/virtual-kubelet/virtual-kubelet/issues/166) to ensure that things like Daemonsets, Pods with hostPath volumes, or other things [not supported by EKS/Fargate](https://docs.aws.amazon.com/eks/latest/userguide/fargate.html#fargate-considerations), don't run on the Fargate Nodes.
**Which service(s) is this request for?**
EKS/Fargate
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
Right now, setting up EKS/Fargate requires a small [amount of AWS-side configuration](https://docs.aws.amazon.com/eks/latest/userguide/fargate-profile.html), and inverts the relationship from "pods are scheduled by their labels" to "a set of labels and namespaces is defined elsewhere to pull pods out of the scheduler and onto Fargate", which is an added complexity when already doing passably-complex setups involving `nodeSelectors` and `taints`/`tolerations`.
**Are you currently working around this issue?**
Not using Fargate at this point. This hasn't exactly encouraged me to experiment with it, since I can't as easily mess about with the scheduler and experiment from kubectl, as I can with other k8s systems like the Cluster Autoscaler.
**Additional context**
Anything else we should know?
**Attachments**
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)
Contributor guide
Research direction
Start with the EKS/Fargate documentation and the linked Virtual Kubelet and AWS Fargate provider references. Review the stated Kubernetes features and unsupported workloads, including DaemonSets and hostPath volumes, to define the required scope. Done means Fargate can interact with node selectors, taints, tolerations, affinity, metrics, and topology spread constraints without admitting unsupported workloads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100