NIM model selection treats a fully-occupied GPU as having 100% free VRAM
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 43m
- Merged PRs (30d)
- 718
Description
## Investigation Summary
- `detectGpu()` (`src/lib/inference/nim.ts`) collapses a genuine zero-free-VRAM `nvidia-smi` reading and an unparseable `[N/A]` reading into the same internal value, then drops `availableMemoryMB` from its result whenever the sum is `0`, for either reason.
- `nimUsableMemoryMB()` (#10122) relies on the opposite: it only falls back to `totalMemoryMB` when `availableMemoryMB` is absent, and has a dedicated test asserting a real `0` is preserved — but `detectGpu()` can never produce that real `0`, so the fallback fires anyway on a saturated GPU.
- Effect: a GPU fully occupied by another workload (`nvidia-smi --query-gpu=memory.free` legitimately reports `0` in that case) is reported as `availableMemoryMB: undefined`, so NIM model selection offers every catalog model as "fitting" on a GPU with no free VRAM.
## Description
When `nvidia-smi` reports a GPU's free memory as genuinely `0`, `detectGpu()` reports it identically to a GPU whose reading failed to parse: `availableMemoryMB` is omitted either way. `nimUsableMemoryMB()` treats an omitted `availableMemoryMB` as "unknown, use total capacity", so a fully-saturated GPU is treated as if 100% of its VRAM were free.
Expected: a genuine `0` should be reported as `availableMemoryMB: 0`, distinct from an unparseable reading (which should stay omitted).
## Reproduction Steps
1. Import `detectGpu` from `src/lib/inference/nim.ts` with the `runCapture` runner mocked to return `"NVIDIA H100 80GB HBM3, 81920, 0\n"` for the `nvidia-smi --query-gpu=name,memory.total,memory.free` call.
2. Call `detectGpu()`.
3. `result.availableMemoryMB` is `undefined` instead of `0`, even though the reading was real, not unparseable.
4. On a host with a GPU fully occupied by another workload, `nemoclaw inference set` would present every NIM model as fitting rather than reporting zero usable memory.
## Environment
- OS: Ubuntu 24.04 LTS (x86_64)
- Hardware: x86_64 workstation (reproduced via direct unit-level execution of `detectGpu()` with a mocked `nvidia-smi` runner; no live NVIDIA GPU needed)
- Node.js: v22.23.1
- Docker: not installed in this environment (not needed — pure detection/parsing logic)
- NemoClaw: reproduced against `main` at commit `02d9198fde7e012257c65b6529a9e05f02a2fb1a`
## Impact
Impact: silent-wrong-result
Who hits this: any user running `nemoclaw inference set` (the
documented NIM model-selection command) on a host with an NVIDIA GPU that is fully occupied by
another workload at probe time. The CLI silently presents catalog models as "fitting" when the
GPU actually has zero free VRAM, so the selected model then fails to load or evicts the
already-running workload — the user gets a wrong recommendation with no error or warning.
Contributor guide
Research direction
Start with `detectGpu()` in `src/lib/inference/nim.ts` and inspect how it parses the `nvidia-smi --query-gpu=name,memory.total,memory.free` output. Reproduce the issue with the mocked output `"NVIDIA H100 80GB HBM3, 81920, 0\n"`. Done means a genuine zero is returned as `availableMemoryMB: 0`, while an unparseable reading remains omitted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 85/100