nebari-dev / nebari-dev/llm-serving-pack

Served models on tainted GPU nodes can't get Longhorn storage

Open
#124 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
3
Forks
3
Avg merge
16h 31m
Merged PRs (30d)
11

Description

Found while testing #117 live on an AWS Nebari cluster.

When GPU nodes carry a taint (e.g. nvidia.com/gpu) and Longhorn's components do not tolerate it, longhorn-manager is not scheduled on the GPU nodes, so they never register as Longhorn nodes and Longhorn cannot attach PVCs there:

FailedAttachVolume: ... unable to attach volume <pvc> to <gpu-node>:
node.longhorn.io "<gpu-node>" not found

The pack defaults model storage to the longhorn StorageClass (defaults.storage.storageClassName), so a served LLMModel scheduled onto a tainted GPU node gets stuck in Init with its model-storage PVC unattached, and never serves.

Workaround (per model): set spec.model.storage.storageClassName to a class that attaches on the GPU node, e.g. gp2/EBS (WaitForFirstConsumer, provisions in the node's AZ).

Suggested fix: document this for GPU node groups, and/or default served-model storage to a node-local class when the model requests a GPU. Not addressed by code yet.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the model storage defaults at defaults.storage.storageClassName and the per-model spec.model.storage.storageClassName setting, along with the GPU-node scheduling behavior described in the issue. Done means documenting the Longhorn and tainted-node limitation for GPU node groups, or resolving and implementing an agreed default-storage behavior for GPU models.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, documentation, infrastructure
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.