NVIDIA / NVIDIA/NemoClaw

Product-scope decision: managed Qwen3.8 27B llama.cpp profile for DGX Spark

Open
#10,112 0 comments 0 reactions 0 assignees View on GitHub
area: local-models area: providers needs: design platform: dgx-spark
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Problem Statement

NemoClaw's managed llama.cpp path ships two DGX Spark profiles (Meta Muse Glimmer 30B, NVIDIA Nemotron 3 Nano 30B-A3B). Operators who want a general-purpose Qwen-family model on a single DGX Spark have no managed option and must run an operator-managed `llama-server` themselves, giving up the pinned-digest acquisition, verified cache, authenticated container, and no-egress runtime contract that the managed path provides.

#9682 implements a Qwen3.8 27B profile. That PR is correctly on a product-scope hold (#9682 review by @apurvvkumaria) because no accepted decision establishes ownership, lifecycle, compatibility, security, and validation expectations for the addition. **This issue exists to record that decision, or to reject the proposal.** It is not a request to merge #9682 as-is.

## Desired Behavior

An operator on a qualified DGX Spark selects a managed Qwen3.8 27B runtime by exact recipe ID:

```bash
NEMOCLAW_PROVIDER=install-llama-cpp \
NEMOCLAW_LLAMACPP_RECIPE=llama-cpp.qwen3-8-27b.spark-single.v1 \
nemoclaw onboard --non-interactive --yes-i-accept-third-party-software
```

**Before:** no managed Qwen option; operator runs and secures `llama-server` themselves.
**After:** NemoClaw acquires the digest-pinned GGUF, verifies size and SHA-256, starts the authenticated container with no egress, and registers `llama-cpp-local`.

## Scope and Exclusions

**Includes:** one `ServingRecipe` + one `ServingPreset`, catalog registration, selection coverage, and guide updates. No new runtime image, no new serving contract, no privilege change — it reuses the llama.cpp source revision, runtime image, CUDA base, and downloader/probe digests already pinned by the Nemotron recipe.

**Excludes:** multimodal projection and speculative decoding, both disabled, though the source repository publishes `mmproj-*` and MTP draft files. No vLLM/NVFP4 companion profile. No change to which profile is recommended — the proposal is `selection: explicit-only`, so it never enters automatic selection or the interactive menu.

**Constraints that materially change the solution — these are the decisions being requested:**

1. **Publisher policy.** The checkpoint is [`unsloth/Qwen3.8-27B-GGUF`](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF) — Apache-2.0, ungated, quantized from `Qwen/Qwen3.8-27B`, but published by a third party rather than `nvidia/`. Precedent exists (`Inferact/Muse-Glimmer-…` ships Experimental). **Is a non-NVIDIA publisher acceptable for a managed profile, and under what conditions?**
2. **Minimum DGX Spark qualification evidence.** **What is the minimum bar for a new Spark profile** — the bounded physical qualification described for Muse Glimmer, the full `llama-cpp-dgx-spark-qualification` contract, or something else?
3. **Reasoning-control mismatch.** `buildLlamaCppServerArguments` emits `--chat-template-kwargs {"reasoning_strength": …}`. This checkpoint's embedded template reads **`reasoning_effort`**, accepts only `xhigh|medium|low`, and raises on other values — so the declared strength is ignored and the template resolves to its own `xhigh` default. The contract's `high` would be invalid for it even if the name matched. **Is that acceptable for this profile (documented, declared value matches observed behavior), or must the contract learn to emit `reasoning_effort` first?** The latter affects every non-Nemotron template, not just this profile.

## Ongoing Ownership

**Proposed:** @zac-wang-nv maintains the recipe, preset, and guide section — digest and revision bumps, quantization changes, and llama.cpp runtime compatibility as the pinned image advances.

**This needs maintainer confirmation.** The scope hold requires one accountable maintainer, and a contributor cannot self-assign that role. If maintainers prefer a different owner, or prefer this live in NemoClaw Community rather than core, that decision should be recorded here.

## Placement and Support Expectations

**Requested placement:** core NemoClaw, `supportState: experimental`, matching the two existing llama.cpp Spark profiles.

**Support expected after merge:** same as the existing experimental profiles — no supported agent/model/runtime tuple until the protected qualification and activation gates pass. If maintainers judge that a third-party-published model should not sit in core at any support state, NemoClaw Community is an acceptable outcome and the PR can be redirected.

## Validation Plan

**Already performed** (#9682, on NVIDIA RTX PRO 6000 Blackwell / sm_120 / x86_64 / driver 595.80 — **not a DGX Spark**):

- Exact-file acquisition at the pinned revision: 17,559,178,144 bytes and SHA-256 `3f22707900…c8b01e`, both matching the recipe pins.
- Server started with the exact argv `buildLlamaCppServerArguments` produces; `n_ctx_slot = 262144`.
- `/v1/models` reports `qwen3-8-27b`, matching `readiness.expectedModel`; `/health` ok.
- Bearer auth enforced on completions (401 with no key and with a wrong key).
- Text round-trip and one structured tool call (`finish_reason: tool_calls`, arguments parse as JSON).
- 32.7 GiB GPU at full context (≈16.4 model + 16 KV).

**Still required, and cannot be produced without the target hardware:**

- Physical DGX Spark GB10 (arm64, sm_121) run — the preset requires arm64 and a qualified `host.platform.dgx_spark`, so the profile cannot even be *selected* on the host used above.
- GB10 unified-memory behavior at `contextSize: 262144`, where the ≈33 GiB comes out of the shared 128 GB pool rather than dedicated VRAM.
- Whichever qualification tier maintainers set in decision (2).

**Requested:** maintainer direction on who can run Spark qualification and against which contract. Happy to run it if given access to a Spark host.

## Compatibility Requirements

| | |
|---|---|
| Hardware | DGX Spark GB10, single node, one NVIDIA GPU, `linux/arm64` |
| Driver | `580.65.06` or later (inherited from the existing preset requirement block, unchanged) |
| Runtime | Docker only. Podman and Kubernetes do not provide the required host-local-inference capability |
| Model context | 262144, the checkpoint's `max_position_embeddings`; 16 full-attention layers of 64 give 64 KiB/token KV |
| Excluded | Any non-Spark platform; multimodal; speculative decoding; model or serving-argument overrides |

## Security or Privacy Impact

No change to permissions, execution boundaries, or credential handling. The profile reuses the existing contract: `policy.egress: disabled`, `modelDownloads: disabled`, `modelSource: verified-local`, container runs as the current non-root user on a Docker internal network with no published port, API key delivered through an owner-only read-only file rather than a process argument, and the Web UI, slot inspection, server tools, agent mode, MCP proxy, router, and multimodal projection all disabled.

**The one new trust element is decision (1):** the model artifact is published by a third party. It is mitigated by pinning the repository revision, the exact filename, the byte size, and the SHA-256 digest, all verified before the runtime starts — so a later re-upload cannot silently change what is served. No new image is introduced.

## Implementation Idea

Already implemented in #9682: one recipe, one preset, three catalog expectation lists, one selection test, and guide updates. If the decisions above land differently — different publisher, different placement, or a `reasoning_effort` contract change first — the PR should be reworked or closed accordingly.

## Category

Provider

## Checklist

- [x] I searched existing issues and this is not a duplicate
- [x] I described the problem and desired behavior

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked #9682 implementation, the existing Muse Glimmer and Nemotron Spark profiles, and the llama-cpp-dgx-spark-qualification contract. Review the three requested decisions about publisher policy, Spark evidence, and reasoning control; done means maintainers record the decisions, confirm ownership and placement, and direct whether #9682 should proceed, change, or close.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker
Domain
ai-infra-agents, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.