[Fargate] [request]: Fargate CPU Scheduling issues
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
When scheduling deployments on EKS Fargate exclusively, pods will sometimes fail to be created. This seems to be because the Fargate scheduler bases its capacity on the 'request' parameter instead of the 'limit', and when the limit is larger than request, this bug is reproduced.
Can Fargate be configured such that it allocates enough capacity for the 'limit' parameter of deployments instead of the 'request' when one is present?
**Which service(s) is this request for?**
Fargate with EKS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
`Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: failed to write "30000": write /sys/fs/cgroup/cpu,cpuacct/kubepods/burstable/podfc67fe40-da5c-4dbc-8989-7be9ac4b5d55/operator/cpu.cfs_quota_us: invalid argument: unknown`
The above error appears in logs (With variations in the value that 'failed to write' and the pod identifier) when attempting to schedule pods on Fargate EKS. The issue is related to CPU requests vs limits, and occurs whenever the 'limit' is higher than the 'request', as is the case on many packaged K8S deployments. While the manual workaround of setting the request to the same as the limit is viable, it is a manual step that doesn't seem like should be necessary.
**Are you currently working around this issue?**
When setting the CPU and memory 'request' parameters in the Kubernetes deployment to match the 'limit' parameters, the pods are able to be scheduled and run. This was confirmed by AWS Support that this is the expected solution.
**Additional context**
This behaviour was seen on multiple, unrelated EKS clusters.
Contributor guide
Research direction
Start by reproducing an EKS Fargate deployment where CPU requests are lower than limits, then inspect the reported containerd/runc cgroup error in the logs. Done means pods schedule and run with differing request and limit values, without requiring requests to be manually raised to match limits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 20/100