awslabs / awslabs/amazon-eks-ami
Support for NVIDIA GPUDirect Storage with EFA
- Dominant language
- Go
- Stars
- 2.7k
- Forks
- 1.2k
- Avg merge
- 2d 46m
- Merged PRs (30d)
- 18
Description
**What would you like to be added**:
Customers serving very large language models (>500GB) need a fast way to load the model into GPU memory as fast as possible. With the [launch](https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-fsx-lustre-elastic-fabric-adapter-nvidia-gpudirect-storage/) of EFA support for FSx for Lustre, a throughput of 1200Gbps can be supported. If I am not mistaken this requires the inclusion of the the [GDS](https://docs.aws.amazon.com/fsx/latest/LustreGuide/configure-efa-clients.html#install-gds-on-client) driver in the AMI.
**Why is this needed**:
Loading large language models into GPU memory is a very time consuming process. Downloading it every time is not feasible. S3, EFS, EBS based options all have considerable performance penalties (e.g. extra copies through a bounce buffer in the CPU’s memory). GPUDirect Storage enables a direct data path between FSxL storage and GPU memory
Contributor guide
Research direction
Start by reviewing the AMI build configuration and the existing EFA support, then compare the requested setup with the FSx for Lustre GDS installation guidance linked in the issue. Done should mean the resulting AMI includes the required GDS driver and supports the stated EFA-based path for loading models from FSx for Lustre.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, linux
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100