[EKS] [request]: command line and API programmatic parameters to control threads per core
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
- [Reference](https://twitter.com/vsoch/status/1731028581432590807) to original tweet with request https://twitter.com/vsoch/status/1731028581432590807
**Tell us about your request**
For high performance computing (HPC) applications we tend to want one thread per core. I know about the great HPC series/family (we use them)! and control with creating single EC2 instances. The case I'd like a handle for that is managed node groups in EKS.
**Which service(s) is this request for?**
EKS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
We are testing using managed node groups with spot instances in EKS, and our current "best" (meaning only) option is to use the logic here: https://aws.amazon.com/blogs/compute/disabling-intel-hyper-threading-technology-on-amazon-linux/ to update the nodes. Since these are spot, this means we need a separate thread or process to:
1. watch for new nodes
2. determine when a new node is added, pause running experiments
3. connect to the node (I am using a kubectl plugin but the other strategy, more complex, is to create a bastion for ssh to the private subset)
4. run the "runtime" example from the link above
5. flag that the group is again ready
The above is a bit janky, because you can imagine that there is nothing stopping completing an operation, and then (between some duration of checking) getting a new node and having a job run without the proper threading! It's very error prone. (this is me running this logic--> 🤪) and I am hoping that we can we have:
1. First priority: API programmatic ability to specify the threads (ideally through functions to [create nodegroup](https://boto3.amazonaws.com/v1/documentation/api/1.26.85/reference/services/eks/client/create_nodegroup.html) and similar)
2. Second priority, the same, but from the command line (we use command line tools less)
both to automate the above and ensure that when a node comes up and it is deemed `Ready` it is also ready from a threading standpoint!
**Are you currently working around this issue?**
See the previous answer! We have a thread running that is keeping a cache of known nodes, and when a new one is found, we pause running experiments and use the [kubectl-node-shell](https://github.com/kvaps/kubectl-node-shell) plugin to issue one-off commands to update the threading. We then re-enable applications. Of course any applications running in the transition state are going to fail.
**Additional context**
I am very happy to test this out and give feedback. And it was really fun to learn how to "hotload" myself! So that's the silver lining I think. Thank you for that! :avocado:

Contributor guide
Research direction
Start with the EKS managed node group API, especially the referenced create_nodegroup operation, and compare the requested parameter with the corresponding command-line interface. Done means managed node groups can be created or configured with a threads-per-core setting that is applied before nodes become Ready, with both programmatic and CLI controls documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100