[ECS] [request]: Support NVidia Multi Instance GPUs (MiG) in task definition
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
I would like ECS to allocate a GPU "slice" instead of the whole GPU when NVidia MIG is enabled. Because slices can be hybrid in VRAM usage across different GPU models and partitioning schemes, it should be smart in alllocating the minimum amount of slices for a certain VRAM capacity, or a slice/GPU that's large enough, similar to how RAM is allocated to containers 1MB at a time.
For example, let's suppose you have many services where each of their tasks occupy 4GB of memory. Since the smallest slice is 5GB, that's okay, you divide a p4 into many 5GB slices. But you might have one that has 8GB, so it's useful to be able to specify that this task demands 8GB to be available, so you either have to use 2 slices or even damand a single larger slice that might be available on other instance type or here. There's that weird way of creating a custom criteria for the task scheduler but it's very obscure.
**Which service(s) is this request for?**
ECS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
The Multi Instance GPU feature for Ampere GPUs has been available for a while. It allows "slicing" the GPU into smaller units and those units are isolated in a hardware level. It's supported in Docker and Kubernetes, which allows you to provide one or more GPU slices to a container, giving it isolation and more granularity.
One of the major problems when working with GPUs in the cloud is the lack of granularity in them, because often
you can fit 4, 8, even 10+ models in a single EC2 instance. You can run multiple processes in a GPU, but they aren't isolated or limited like the usual CPU/RAM allocation of a container. Right now on ECS you either share the GPU across all containers or you have to give the entire GPU to one of them. ECS also doesn't know how much VRAM a container wants, so it creates a dependency between the TaskDefinition and the EC2 instances available in the clusters, because the smallest GPU unit might not fit one or other models that are unusually large.
The new EC2 instances with Ampere support greater granularity and it would help optimize costs/performance if we could use that.
**Are you currently working around this issue?**
Sharing the same GPU across all containers, and developers have to know how much VRAM they will use at most. The container's CPU and RAM requirements are created in such a way that containers theoretically should never allocate VRAM to the point of exhausting it, as long as the cluster instances are certain specific types.
This comes with issues, like the CPU and RAM requirement of the task definition being tied to the GPU requirement in a strange way, and a container being able to see what the others are doing in the GPU and even crash them. There are no guarantees that a container actually respects its share of VRAM. The instance types that we can use also end up limited to be ones that the CPU/RAM math workaround works.
Contributor guide
Assessment
This issue has not been assessed yet.