Design cloud-agnostic model storage for LLM serving

Problem

Model downloads need persistent storage that is:

• Independent of node ephemeral disk size (models can be 5GB-200GB+)
• Persistent across pod restarts to avoid re-downloading
• Eventually multi-cloud (AWS, GCP, Azure, k3s)

Currently, emptyDir fails on nodes with limited disk (e.g. g6.xlarge has ~20GB). Longhorn PVCs are limited by node disk. We need storage backed by cloud-native block or file storage.

Current state

• defaults.storage.storageClassName is now configurable via Helm values (just merged)
• Per-model storageClassName override already exists in the LLMModel CR
• The operator applies the default when a model doesn't specify one

Storage options by cloud

Design considerations

Single-replica models

Block storage (EBS, PD, Azure Disk) works well. Fast, cost-effective, and the PVC persists across pod restarts.

Multi-replica models sharing downloads

Block storage is RWO - each replica needs its own PVC and download. Options:

1. File storage (RWX) - one download shared by all replicas. Slower but cheaper.
2. ReadOnlyMany from snapshot - download once to RWO, snapshot, mount as ROX to replicas.
3. OCI modelcar images - pre-package model into an OCI image, no download needed.

What nebari-infrastructure-core needs to provide

• EBS CSI driver addon enabled when GPU node groups exist (AWS)
• A StorageClass (e.g. ebs-gp3) created and passed to the LLM serving pack
• Equivalent for GCP/Azure when those providers are implemented

Open questions

• Should the LLM serving pack have an opinion about storage class, or always defer to what's configured?
• For multi-replica, should we default to RWX (simpler) or RWO-per-replica (faster)?
• Should model downloads be decoupled from pod lifecycle entirely (e.g., a separate download Job that populates the PVC)?

Start by reviewing the existing defaults.storage.storageClassName, the per-model storageClassName override in the LLMModel CR, and how the operator applies them. Compare the listed block, file, snapshot, and OCI options, then resolve the open questions and define what nebari-infrastructure-core and the LLM serving pack must provide for each supported cloud.