aws / aws/containers-roadmap

[EKS] [request]: command line and API programmatic parameters to control threads per core

Open
#2,225 4 comments 3 reactions 0 assignees View on GitHub
EKS EKS Managed Nodes Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

- [Reference](https://twitter.com/vsoch/status/1731028581432590807) to original tweet with request https://twitter.com/vsoch/status/1731028581432590807

**Tell us about your request**

For high performance computing (HPC) applications we tend to want one thread per core. I know about the great HPC series/family (we use them)! and control with creating single EC2 instances. The case I'd like a handle for that is managed node groups in EKS.

**Which service(s) is this request for?**

EKS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**

We are testing using managed node groups with spot instances in EKS, and our current "best" (meaning only) option is to use the logic here: https://aws.amazon.com/blogs/compute/disabling-intel-hyper-threading-technology-on-amazon-linux/ to update the nodes. Since these are spot, this means we need a separate thread or process to:

1. watch for new nodes
2. determine when a new node is added, pause running experiments
3. connect to the node (I am using a kubectl plugin but the other strategy, more complex, is to create a bastion for ssh to the private subset)
4. run the "runtime" example from the link above
5. flag that the group is again ready

The above is a bit janky, because you can imagine that there is nothing stopping completing an operation, and then (between some duration of checking) getting a new node and having a job run without the proper threading! It's very error prone. (this is me running this logic--> 🤪) and I am hoping that we can we have:

1. First priority: API programmatic ability to specify the threads (ideally through functions to [create nodegroup](https://boto3.amazonaws.com/v1/documentation/api/1.26.85/reference/services/eks/client/create_nodegroup.html) and similar)
2. Second priority, the same, but from the command line (we use command line tools less)

both to automate the above and ensure that when a node comes up and it is deemed `Ready` it is also ready from a threading standpoint!

**Are you currently working around this issue?**

See the previous answer! We have a thread running that is keeping a cache of known nodes, and when a new one is found, we pause running experiments and use the [kubectl-node-shell](https://github.com/kvaps/kubectl-node-shell) plugin to issue one-off commands to update the threading. We then re-enable applications. Of course any applications running in the transition state are going to fail.

**Additional context**

I am very happy to test this out and give feedback. And it was really fun to learn how to "hotload" myself! So that's the silver lining I think. Thank you for that! :avocado:

![image](https://github.com/aws/containers-roadmap/assets/814322/972b82de-d7e5-4d99-b524-59071f12e1c9)

Contributor guide

Open the contributing guide

Research direction

Start with the EKS managed node group API, especially the referenced create_nodegroup operation, and compare the requested parameter with the corresponding command-line interface. Done means managed node groups can be created or configured with a threads-per-core setting that is applied before nodes become Ready, with both programmatic and CLI controls documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.