Track in-stack GPU inference support (vLLM / llama.cpp across NVIDIA, ROCm, Apple)

Background

Follow-up to https://github.com/ObolNetwork/obol-stack/pull/422.

PR #422 added discovery/registration of host-local OpenAI-compatible inference endpoints. That is useful as a compatibility/fallback layer, but it does not make vLLM, llama.cpp, or similar inference runtimes first-class Kubernetes workloads inside the Obol stack.

We want to track whether/how Obol can run inference servers inside the stack while preserving hardware acceleration and making resource contention visible to Kubernetes.

Feasibility matrix

Target	Best runtime	Current feasibility	Notes
NVIDIA / Spark-class hosts	vLLM	Medium	Runtime/model works in host Docker. Existing k3d/K3s cluster cannot schedule GPU pods until NVIDIA runtime is configured inside the k3d node/K3s containerd.
NVIDIA / Spark-class hosts	llama.cpp CUDA	Medium	Same Kubernetes GPU runtime blocker. Less compelling than vLLM on NVIDIA, but useful as a fallback.
AMD ROCm / SilverMesh-class hosts	llama.cpp ROCm	High	k3d can expose AMD GPU devices; ROCm device plugin registered amd.com/gpu: 1 in a throwaway cluster. Need image/build strategy + Deployment wiring.
AMD ROCm / consumer Navi/gfx1030	vLLM ROCm	Low / experimental	Not the first target. llama.cpp ROCm is the practical path.
Apple Silicon	llama.cpp Metal / MLX / LM Studio	Host-only	Linux containers/k3d do not get Metal/MLX acceleration. Keep host-native runtime and use PR #422 discovery.

Findings from probes

PR #422 / host endpoint discovery

• Host-local vLLM on port 8000 was discovered and mapped for k3d pod access via http://host.k3d.internal:8000.
• A pod inside the llm namespace could reach http://host.k3d.internal:8000/v1/models successfully.
• Actual merged env vars are:
  • OBOL_DISABLE_LOCAL_MODEL_DISCOVERY=true
  • OBOL_LOCAL_MODEL_DISCOVERY_PORTS=9000:vllm,5001:custom
• Follow-up bug: vLLM can be cosmetically labeled llama-server@8000 because the detector treats /health success as llama.cpp. vLLM also serves /health. Routing still works, but runtime metadata becomes misleading.

AMD ROCm / k3d

A throwaway k3d cluster with:

--volume /dev/kfd:/dev/kfd@server:0
--volume /dev/dri:/dev/dri@server:0

plus the ROCm Kubernetes device plugin successfully exposed:

amd.com/gpu: 1

Device plugin logs showed registration of amd.com_gpu and one AMD GPU discovered. This proves the Kubernetes device scheduling layer is feasible for in-stack llama.cpp ROCm.

Open item: a full ghcr.io/ggml-org/llama.cpp:server-rocm smoke did not complete because the upstream image has a large ROCm layer. A follow-up host pull partially downloaded layers, the SSH session exited, and the orphaned remote pull was stopped. The image was not present afterward. We need a supervised pre-pull/import or slimmer image strategy.

NVIDIA / k3d

Host Docker GPU inference works, but a CUDA pod smoke in the existing k3d cluster failed with:

no runtime for "nvidia" is configured

Interpretation:

• Host Docker can use the GPU via Docker DeviceRequests.
• k3d node containers may see /dev/nvidia*.
• That is not enough: K3s/containerd inside the node needs a configured NVIDIA runtime.
• Stock rancher/k3s node images do not include nvidia-container-runtime.
• K3s detects NVIDIA runtime at startup only if the runtime binary is available in its PATH.

Remaining work

1. Patch local discovery follow-ups

• Fix runtime/server-type detection so vLLM is not mislabeled as llama.cpp just because /health works.
  • Prefer stronger llama.cpp signals such as /props, response body, or explicit port label.
  • Prefer vLLM signals such as /version, model metadata, or explicit port label.
• Update docs/tests to use the merged env var names:
  • OBOL_DISABLE_LOCAL_MODEL_DISCOVERY
  • OBOL_LOCAL_MODEL_DISCOVERY_PORTS
• Add tests covering vLLM-compatible /health behavior.

2. Add GPU capability/status preflight

Potential UX:

obol model gpu status
obol model discover --cluster

Should report:

• Host GPU vendor: amd, nvidia, apple, none.
• Cluster backend: k3d vs k3s.
• k3d config requirements present/missing.
• Kubernetes node extended resources: amd.com/gpu, nvidia.com/gpu.
• RuntimeClass availability and NVIDIA smoke status.
• Host-local endpoints and in-cluster inference Services.

3. Add generated GPU profiles for k3d/k3s

AMD ROCm k3d profile:

• Mount /dev/kfd.
• Mount /dev/dri.
• Install/apply ROCm device plugin.
• Verify amd.com/gpu is allocatable before deploying inference.

NVIDIA k3d profile:

• Build/publish an Obol-compatible K3s node image that contains NVIDIA runtime tooling.
• Create k3d cluster with options.runtime.gpuRequest: all / --gpus all.
• Install/apply NVIDIA device plugin.
• Ensure GPU pods can use runtimeClassName: nvidia and request nvidia.com/gpu: 1.
• Gate vLLM deployment on a passing CUDA smoke.

Direct k3s alternative:

• Evaluate direct --backend k3s on GPU hosts where sudo is available.
• Confirm host K3s detects /usr/bin/nvidia-container-runtime correctly.
• Document sudo/passwordless-sudo requirement clearly.

4. Add first-class in-stack inference deployments

Add a command/config path for deploying an inference runtime as a Kubernetes workload, e.g.:

obol model deploy llama-cpp \
  --name mesh-llama-rocm \
  --model-path /path/to/model.gguf \
  --gpu amd \
  --runtime llama.cpp

obol model deploy vllm \
  --name qwen36-fast \
  --model-path /path/to/model \
  --draft-model-path /path/to/draft-model \
  --gpu nvidia

Implementation checklist:

• Deployment/StatefulSet template for llama.cpp server.
• Deployment/StatefulSet template for vLLM.
• Model storage config: hostPath for local dev, PV/PVC later.
• Service in llm namespace, e.g. llama-cpp.llm.svc.cluster.local:8080.
• LiteLLM config entry pointing at the in-cluster Service.
• Long startup probes for model loading.
• Readiness probes that verify OpenAI-compatible /v1/models or /health.
• CPU/memory/GPU resource requests and limits.

5. SilverMesh/ROCm next test

• Decide image strategy:
  • supervised pre-pull/import of ghcr.io/ggml-org/llama.cpp:server-rocm, or
  • local slimmer image based on the existing ROCm-enabled llama-server build.
• Run minimal model smoke in a throwaway k3d cluster.
• Verify:
  • pod schedules with amd.com/gpu: 1,
  • model loads,
  • /v1/models works,
  • a small /v1/chat/completions request works,
  • GPU VRAM/busy metrics show real ROCm usage.

6. Spark/NVIDIA next test

• Choose path: direct k3s with sudo vs custom k3d NVIDIA node image.
• Run CUDA smoke in a throwaway cluster.
• Only after CUDA smoke passes, deploy vLLM as an in-cluster workload.
• Verify host external vLLM can be disabled without breaking LiteLLM routing.

7. Resource policy

Putting inference inside Kubernetes does not remove contention; it makes contention visible and enforceable.

• Enforce resources.limits.{amd.com/gpu,nvidia.com/gpu}: 1 on single-GPU hosts.
• Set CPU/memory requests and limits for vLLM/llama.cpp.
• Avoid simultaneously running external Docker/systemd inference and in-cluster inference against the same GPU.
• Add node labels/taints such as obol.ai/gpu=true if needed.
• Default to one model-serving pod per single-GPU host unless MIG/time-slicing is explicitly configured.

8. Apple Silicon later

• Treat Apple acceleration as host-native only.
• Use PR #422 discovery for llama.cpp Metal, MLX, LM Studio, or similar OpenAI-compatible host servers.
• Document that k3d/Linux pods on macOS do not get Metal/MLX acceleration.

Acceptance criteria

• Host endpoint discovery remains the reliable fallback path.
• ROCm k3d smoke can schedule a GPU pod and serve a minimal llama.cpp model inside the cluster.
• NVIDIA k3d/k3s smoke can run a CUDA pod before attempting in-cluster vLLM.
• In-cluster inference Services are registered into LiteLLM and usable by buyers/sellers.
• Documentation clearly explains hardware-specific paths and limitations.

Start by reading PR #422 and the GPU feasibility findings, then choose one scoped path from the remaining-work checklist; no implementation files or tests are named. Done requires the selected GPU smoke and deployment path to meet its stated acceptance checks, including scheduling, model serving, and LiteLLM registration where applicable.