nebari-dev / nebari-dev/llm-serving-pack
Served models on tainted GPU nodes can't get Longhorn storage
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 3
- Forks
- 3
- Avg merge
- 16h 31m
- Merged PRs (30d)
- 11
Description
Found while testing #117 live on an AWS Nebari cluster.
When GPU nodes carry a taint (e.g. nvidia.com/gpu) and Longhorn's components do not tolerate it, longhorn-manager is not scheduled on the GPU nodes, so they never register as Longhorn nodes and Longhorn cannot attach PVCs there:
FailedAttachVolume: ... unable to attach volume <pvc> to <gpu-node>:
node.longhorn.io "<gpu-node>" not found
The pack defaults model storage to the longhorn StorageClass (defaults.storage.storageClassName), so a served LLMModel scheduled onto a tainted GPU node gets stuck in Init with its model-storage PVC unattached, and never serves.
Workaround (per model): set spec.model.storage.storageClassName to a class that attaches on the GPU node, e.g. gp2/EBS (WaitForFirstConsumer, provisions in the node's AZ).
Suggested fix: document this for GPU node groups, and/or default served-model storage to a node-local class when the model requests a GPU. Not addressed by code yet.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the model storage defaults at defaults.storage.storageClassName and the per-model spec.model.storage.storageClassName setting, along with the GPU-node scheduling behavior described in the issue. Done means documenting the Longhorn and tainted-node limitation for GPU node groups, or resolving and implementing an agreed default-storage behavior for GPU models.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, documentation, infrastructure
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100