Model Lab: connect CodeWhale traces to open-weight fine-tuning/eval services

## Goal

Let users turn their own CodeWhale coding sessions into curated datasets, eval suites, and optional fine-tuning jobs for open-weight models. CodeWhale becomes the harness where people can improve open models, not just consume them.

## Where this fits in the stack

- **CodeWhale** is the workbench — runs the agent, owns the traces, owns the consent surface.
- **Providers** are where models run (DeepSeek, Hugging Face Inference Providers, OpenRouter, Novita, Fireworks, Together, Hyperbolic, SiliconFlow, NVIDIA NIM, plus local: vLLM, SGLang, Ollama, llama.cpp, TGI).
- **Worksets** are curated optional capability packs — they extend the lab without bloating core.
- **Fin** is the cheap evaluator / router / verifier seam (Goal mode in v0.8.43, generalized verifier preview in v0.8.46, eval pipeline in v0.9.0).
- **Goal mode** (v0.8.43, #1976) is the long-running loop that turns model changes into measured improvements.

Model Lab is the user-facing surface that sits on top of all of that. It turns the workbench's accumulated traces into datasets, runs eval replays, hands curated data to a fine-tuning backend, and integrates open-source worksets without vendoring them into core.

## Non-goals

- **No automatic upload.** Nothing leaves the user's machine unless they explicitly ran an export command.
- **No hidden telemetry.** Each workset declares its telemetry posture; off by default.
- **No silent hosted routing.** Provider / backend selection is always explicit.
- **No "download random model and trust it."** Every model installed to the lab goes through an explicit install with license + provenance shown.
- **No proprietary-model-first workflow.** Open-weight models are the first-class target.
- **No provider lock-in.** Hosted backends are pluggable; the export format is provider-neutral.
- **No vendoring of heavy GPU / Python / NVIDIA dependencies into CodeWhale core.** Worksets install on-demand.

## Provider vs Workset — the architectural split

- **Providers** live in CodeWhale's provider abstraction (v0.8.47 work). They are where models run. They are not optional; they are how the agent talks to a model. Each one has a config block, an auth path, and parity coverage in v0.8.45.
- **Worksets** live in Model Lab. They are optional capability packs that bring open-source ML tooling into the lab loop. Each one installs on-demand and declares license, telemetry posture, network egress, GPU / Python deps, and what data leaves the machine if any.

**Hugging Face is BOTH** — it is a first-class provider (Inference Providers / Router) AND the open-model registry workset (Hub API, model cards, datasets, adapters, Safetensors, Jobs). The two roles ship through different surfaces; they share auth (`HF_TOKEN` / `HUGGINGFACE_API_KEY` alias).

## Architecture: Worksets as the integration shape

CodeWhale core stays lean. Worksets are curated optional packs.

```
/model-lab worksets list # show installed + available
/model-lab worksets install hf
/model-lab worksets install unsloth
/model-lab worksets install nemo
/model-lab worksets install arcee
/model-lab worksets install serving
/model-lab worksets install eval
/model-lab worksets install observability
/model-lab worksets install training-infra
/model-lab worksets remove
/model-lab worksets info # license, telemetry, network, deps
```

## Initial worksets

### Hugging Face Workset

The open-model operating system. Even though HF is also a first-class **provider** for inference (lands in v0.8.47), the **workset** is the registry / dataset / adapter / model-card layer.

- **Model discovery**: search open-weight coding models, filter by license, base model, quantization, tool/reasoning support.
- **Model passport**: show license, base model, context length, chat template, tool-call support, reasoning support, eval results, gated / private status. Pulls from HF Hub API + model cards.
- **Hub integration**: private / public model repos, adapter upload, dataset upload, model-card generation.
- **Dataset path**: export CodeWhale transcripts → redact → dataset → train / eval.
- **Runtime bridges**: Transformers, PEFT, TRL, Accelerate, Safetensors, Inference Providers, LightEval.
- **HF Jobs** (later): compute workflows for training/eval runs initiated from the lab.

Auth: `HF_TOKEN` (canonical) or `HUGGINGFACE_API_KEY` (alias). No upload without an explicit command. Boundary: gated models require explicit consent flow before any pull.

Docs: https://huggingface.co/docs/inference-providers · https://huggingface.co/docs/hub/main/api · https://huggingface.co/docs/hub/model-cards · https://huggingface.co/docs/hub/jobs

### Unsloth Workset

Local fine-tuning, oriented around the "make Brother Whale better at my workflow" path.

- SFT / LoRA / QLoRA as the first simple path.
- Later: DPO / GRPO-style preference-based improvement loops.
- Fin as preflight: checks dataset shape, license / provenance, VRAM estimate, model compatibility, and whether the result is actually deployable.
- Output adapters can be pushed to the user's HF repo via the HF Workset.

Docs: https://unsloth.ai/docs

### NeMo Workset (NVIDIA NeMo, Apache-2.0)

- **Data Designer** — synthetic dataset generation from seed traces (https://github.com/NVIDIA-NeMo/DataDesigner). Telemetry opt-in per their README.
- **Curator** — PII / secret redaction (`PiiModifier` over email, person, phone, URL, location). Useful before any trace export leaves the machine.
- **Evaluator** — reproducible model evaluation; OpenAI-compatible API; coding benchmarks; function-calling; long-context; agentic suites (https://github.com/NVIDIA-NeMo/Evaluator).
- **Guardrails** — optional policy rails on prompt injection, tool-use boundaries, data export (https://github.com/NVIDIA-NeMo/Guardrails).
- **Aligner** — later-stage post-training path (https://github.com/NVIDIA/NeMo-Aligner). Not a first-run default.

Boundary: no traces sent to NVIDIA Build or any hosted endpoint by default.

### Arcee Workset (Arcee open-source catalog)

The "make your own model" path — merge, distill, evaluate, serve open-weight checkpoints.

- **Trinity model family** — open-weight multi-turn / tool-use / structured-output (`arcee-ai/Trinity-Large-Thinking`, `Trinity-Large-Preview`, `Trinity-Mini`, `Trinity-Nano-Preview`).
- **MergeKit** (https://github.com/arcee-ai/mergekit) — model-merging library. Build "my coding agent blend" from open checkpoints, run through CodeWhale evals.
- **DistillKit** (https://github.com/arcee-ai/DistillKit) — distillation toolkit.
- **Spectrum + Arcee Fusion** — advanced merge methods as recipes.
- **Coder / Caller / Maestro / Blitz lines** — verify license + availability before first-class.

```
/model-lab arcee merge
/model-lab arcee distill
/model-lab arcee eval
/model-lab arcee compare
```

### Serving Workset

Local runtime recipes for self-hosted serving.

- **vLLM** — high-throughput batched serving.
- **SGLang** — structured generation, good for tool-use workloads.
- **TGI** (Text Generation Inference) — HuggingFace's serving stack.
- **llama.cpp / Ollama** — CPU / consumer-GPU path.

Each gets a recipe: install + config + smoke test + which CodeWhale provider config to use to talk to it.

### Eval Workset

Reproducible eval harnesses and CodeWhale-specific replay evals.

- **SWE-bench** — software-engineering benchmark.
- **Terminal-Bench** — terminal-agent benchmark.
- **BFCL** — Berkeley Function-Calling Leaderboard.
- **LiveCodeBench** — live coding benchmark.
- **CodeWhale replay evals** — exported traces become eval suites; baseline-vs-finetune comparisons.

### Observability Workset

Trace export and analysis sinks (all opt-in).

- Per-turn cost / latency / reasoning-token / tool-call / failure stats.
- Phoenix-style local trace UI.
- Opik / Langfuse-style export adapters (later).
- Boundary: opt-in per sink, redaction review mandatory before any external sink.

### Training Infra Workset

Hosted execution adapters for fine-tuning / eval runs.

- **Prime Intellect** — distributed fine-tuning.
- **Tinker** — managed fine-tuning service.
- **HF Jobs** — compute workflows on the HF Hub.
- **RunPod / Lambda-style** — on-demand GPU rental (later).

```
/model-lab finetune --provider prime-intellect
/model-lab finetune --provider tinker
/model-lab finetune --provider hf-jobs
```

## Core surface (CodeWhale-native, no workset required)

```
/model-lab capture # mark current session as a candidate trace
/model-lab export # selected sessions → redacted JSONL on disk
/model-lab redact # local redaction review before any upload
/model-lab dataset --from successful-turns
/model-lab eval --against current
/model-lab promote --if-better # only swap default model if eval clears bar
```

Each session export includes prompts, tool calls, diffs, test results, approvals, failures, and final outcome labels — enough to reconstruct the task and grade it. Format is provider-neutral.

## Strong-shape future flow

```
/model-lab capture
/model-lab redact
/model-lab dataset --from successful-turns
/model-lab finetune --workset unsloth --base qwen-or-deepseek-or-glm
/model-lab eval --against current
/model-lab promote --if-better
```

## Dependencies (what unblocks this work)

- **Goal mode + Fin wakeup** (v0.8.43, #1976) — the loop that produces measurable outcomes.
- **HF as first-class provider** (v0.8.47) — the provider abstraction that makes HF Inference Providers / Router a real config block.
- **Verifier preview** (v0.8.46) — generalizes Fin to all claim-of-done events.
- **Eval pipeline** (v0.9.0 sub-project B) — the measurement substrate that grades outcomes.

Until those land, Model Lab is design-only. After they land, the export → redact → eval → finetune surface (plus worksets) becomes implementable.

## Sequencing

- **v0.8.43 → v0.9.0**: stabilize the substrate. No Model Lab implementation yet; this issue stays as living design.
- **Post-v0.9.0 (target v0.10.0)**: scope a dedicated Model Lab release. First wave: HF + Unsloth + NeMo + Arcee + Serving + Eval. Hosted backends: Prime Intellect + Tinker + HF Jobs.
- **v0.10.x**: Observability + Training Infra worksets + community-contributed adapters.

## Safety boundary recap

- Exports require an explicit command — no background telemetry, no session shipping, no auto-fine-tune.
- Redaction review is mandatory before any upload.
- Provider adapters and worksets are opt-in per backend / per pack.
- Each workset declares license, telemetry, network egress, and dependency footprint at install time.
- Gated HF models require explicit consent before any pull.

## Related

- #1976 (v0.8.43) Goal mode — the loop this depends on.
- v0.8.46 verifier preview — the generalized Fin path.
- v0.8.47 milestone — HF as first-class provider lands here.
- v0.9.0 milestone — the eval pipeline that grades the outcomes.