NVIDIA / NVIDIA/NemoClaw

[Linux][Inference] Add generic managed-vLLM profiles for Muse Glimmer and Nemotron 3.5 Lightning

Open
#9,643 0 comments 0 reactions 0 assignees View on GitHub
area: inference area: local-models area: providers needs: design platform: linux provider: vllm
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Summary

Decide whether NemoClaw should add explicit, experimental managed-vLLM profiles for Muse Glimmer 30B and NVIDIA Nemotron 3.5 Lightning on generic Linux hosts with NVIDIA GPUs.

PR #9537 exposed the current behavior: an explicit `NEMOCLAW_VLLM_MODEL` selection stops when the catalog has no compatible Linux preset, even when the model can have a compatible runtime recipe.

## Current behavior

- DGX Spark has explicit Muse Glimmer and Lightning profiles.
- Generic Linux has no matching presets for these models.
- The resolver stops before download or runtime mutation.
- `nemoclaw profiles list` therefore reports no compatible Linux profile for these selections.

## Proposed product decision

Authorize implementation of two explicit-only, Experimental Linux profiles. Do not change any automatic platform default.

Each profile must use the accepted managed-inference catalog and resolver. A new model or host profile must be a catalog change, not a model-specific CLI branch.

## Compatibility boundary

- Linux amd64 and Linux arm64 require separate runtime recipes and immutable image digests.
- Muse Glimmer must not reuse an ARM64-only candidate image on amd64.
- Lightning on Ampere must use its qualified W4A16 execution contract, including the required MoE backend.
- Host admission must evaluate architecture, compute capability, GPU memory, runtime availability, and recipe readiness before download.
- A profile without an accepted runtime and hardware evidence must remain unavailable.

## Lifecycle and security

- Reuse the existing managed vLLM ownership receipt, bearer authentication, loopback listener, route, recovery, and cleanup contracts.
- Keep Hugging Face credentials out of command output, logs, receipts, and sandbox state.
- An explicit incompatible selection must stop before acquisition, container creation, route mutation, or sandbox mutation.
- Existing DGX Spark and DGX Station defaults must not change.

## Validation boundary

- Catalog schema and reference validation.
- Resolver tests for explicit match, incompatibility, ambiguity, and no-match behavior.
- `nemoclaw profiles list` output and incompatibility reason tests.
- Managed vLLM lifecycle and recovery tests.
- Physical evidence for every architecture, GPU family, model, runtime image, and agent combination claimed as tested.
- Brev evidence may establish one tested Linux environment. It does not establish general Linux support.

## Acceptance criteria

- [ ] Maintainers record ownership, lifecycle, compatibility, security, and validation expectations.
- [ ] Maintainers approve the initial Linux architecture and GPU matrix.
- [ ] Every enabled profile pins the model revision and runtime image digest.
- [ ] Explicit selection resolves through catalog data without a model-specific code branch.
- [ ] Unsupported or unqualified combinations stop before effects.
- [ ] Automatic defaults remain unchanged.
- [ ] Documentation distinguishes Experimental, tested, compatible, and supported states.

Related: #8379, #8384, #8385, and PR #9537.

Signed-off-by: Prekshi Vyas

Contributor guide

Open the contributing guide

Research direction

Start by reading PR #9537 and the existing managed-inference catalog, catalog schema validation, and resolver tests. Trace how explicit selections reach `nemoclaw profiles list`, then review the managed vLLM lifecycle and recovery tests. Done means approved architecture and GPU coverage, pinned model and image data, pre-effect rejection of unsupported combinations, unchanged defaults, and updated state documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, typescript
Domain
ai, infrastructure, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.