Document using GDS with EBS on AWS EC2/SageMaker

On AWS Nitro systems it appears that EBS volumes are exposed to the system as NVMe block disks. This means we could potentially use GDS to load data directly into GPU memory.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nvme-ebs-volumes.html

Part 1

We should launch an EC2 instance on Nitro and see if we can get drivers and things configured correctly to get GDS working and reading from the EBS volumes. If successful we should add a guide on how to do this to the docs.

Part 2

This Stack Overflow question also suggests that SageMaker can use the EBS-backed volumes during training.

If it is possible to use GDS with EBS on Nitro we should see if there is a way to get this working for SageMaker batch jobs. It may prove more difficult if it requires driver setup, reboots or anything like that because the VMs for training jobs are launched on your behalf and you can't access them directly.

Start with the linked AWS Nitro, NVMe EBS, and GDS documentation, then test whether GDS can read EBS-backed NVMe volumes on an EC2 Nitro instance. If that works, investigate whether the same setup is possible for SageMaker batch jobs; done means documenting the verified EC2 procedure and any SageMaker findings.