aws / aws/containers-roadmap

[EKS] [request]: Document NFS / network file system backend usage for Run:ai Model Streamer in the EKS AI/ML inference documentation

Open
#2,845 0 comments 17 reactions 0 assignees View on GitHub
EKS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**

Please document the NFS / network file system backend path for the Run:ai Model Streamer in the EKS "Load and Serve Models" documentation (https://docs.aws.amazon.com/eks/latest/userguide/ml-inference-load-serve-model.html). Today, this page only documents the S3-to-GPU streaming path, even though the upstream Run:ai Model Streamer supports "network file systems, S3, Disk, etc." (https://github.com/run-ai/runai-model-streamer/blob/master/docs/README.md).

Specifically, we are requesting the following four items:

1. Add a note on the EKS "Load and Serve Models" page indicating that the Run:ai Model Streamer supports network file systems in addition to S3. The upstream Model Streamer documentation already states it supports "network file systems, S3, Disk, etc.", but the EKS page only documents the S3 path, so readers reasonably conclude that S3 is the only option.
2. Provide a configuration example showing vLLM with --load-format runai_streamer against an NFS PersistentVolume, for example an FSx for ONTAP volume mounted at /models.
3. If available, include benchmark data comparing Model Streamer on NFS versus S3, to help customers with storage selection.
4. Consider adding an NFS mount helper to the vLLM AWS Deep Learning Container (DLC) that auto-configures a /models mount point when the NFS_SERVER and NFS_PATH environment variables are supplied.

**Which service(s) is this request for?**

EKS (with the vLLM AWS Deep Learning Container and Run:ai Model Streamer)

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**

A common setup is a permission-aware RAG system on Amazon EKS where FSx for ONTAP is used as a single, unified storage platform. In this pattern, the model weights sit on the same ONTAP volume that serves documents through S3 access points, and multiple vLLM pods share those weights through an NFS ReadOnlyMany PersistentVolume rather than each pod pulling the weights from S3.

The Run:ai Model Streamer's concurrent tensor loading feature should significantly benefit this NFS-based loading path. However, the EKS documentation only demonstrates the S3 path (and the closest community example uses FSx for Lustre), so there is no documented configuration for the NFS / FSx for ONTAP scenario. As a result, users on NFS-based storage have to infer the correct streamer configuration and concurrency settings themselves, which makes this harder than it needs to be and slows adoption of the shared-volume model-loading pattern.

**Are you currently working around this issue?**

The current workaround is to adapt the streamer configuration and concurrency settings from the S3 and FSx for Lustre examples that are documented today, and apply them to an NFS ReadOnlyMany PV setup on FSx for ONTAP. This works but requires trial and error, since the configuration is not documented for the NFS backend.

**Additional context**

- Upstream Run:ai Model Streamer documentation (lists supported storage types including network file systems): https://github.com/run-ai/runai-model-streamer/blob/master/docs/README.md
- EKS "Load and Serve Models" documentation (currently S3-only): https://docs.aws.amazon.com/eks/latest/userguide/ml-inference-load-serve-model.html
- Related AWS re:Post community article using FSx for Lustre with the Run:ai Model Streamer (a useful reference, but not NFS-based): https://repost.aws/articles/AR7P2P1LpBTQuAqWJPjN-odg

**Attachments**
N/A

Contributor guide

Open the contributing guide

Research direction

Start with the EKS “Load and Serve Models” page and the upstream Run:ai Model Streamer README, then compare the existing S3 and FSx for Lustre examples. Done means the page explains NFS support and includes a vLLM example using an NFS PersistentVolume; benchmark data and an NFS mount helper are optional requests to verify separately.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, documentation, machine-learning
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.