awslabs / awslabs/amazon-eks-ami

Support for NVIDIA GPUDirect Storage with EFA

Open
#2,181 3 comments 6 reactions 0 assignees View on GitHub
Stale
Dominant language
Go
Stars
2.7k
Forks
1.2k
Avg merge
2d 46m
Merged PRs (30d)
18

Description

**What would you like to be added**:

Customers serving very large language models (>500GB) need a fast way to load the model into GPU memory as fast as possible. With the [launch](https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-fsx-lustre-elastic-fabric-adapter-nvidia-gpudirect-storage/) of EFA support for FSx for Lustre, a throughput of 1200Gbps can be supported. If I am not mistaken this requires the inclusion of the the [GDS](https://docs.aws.amazon.com/fsx/latest/LustreGuide/configure-efa-clients.html#install-gds-on-client) driver in the AMI.

**Why is this needed**:

Loading large language models into GPU memory is a very time consuming process. Downloading it every time is not feasible. S3, EFS, EBS based options all have considerable performance penalties (e.g. extra copies through a bounce buffer in the CPU’s memory). GPUDirect Storage enables a direct data path between FSxL storage and GPU memory

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the AMI build configuration and the existing EFA support, then compare the requested setup with the FSx for Lustre GDS installation guidance linked in the issue. Done should mean the resulting AMI includes the required GDS driver and supports the stated EFA-based path for loading models from FSx for Lustre.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, linux
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.