Vijay — fire H200 training in parallel as faster-finish insurance

@beingvijay (or whichever GitHub handle you use)

## TL;DR

Anshuman is running Phase 2 training on **`a10g-large`** right now — slow but stable, finishing ~**3:30 PM IST**. We have only **~1.5h cushion** before the 5 PM deadline.

**Your job: fire the SAME training on `h200` (3× faster) using your $30 HF credit, in parallel.** If H200 succeeds, training finishes by ~12:30 PM and we get **3 hours of cushion** for plot/video/submission. If H200 fails, Anshuman's a10g run is the safety net — no harm done.

This is an insurance bet, not a replacement.

---

## Where we are right now

- ✅ **Currently running**: Phase 2 v6 on `a10g-large`, job ID `69ed99a9d2c8bd8662bcf2ef` ([job page](https://huggingface.co/jobs/anshumanatrey/69ed99a9d2c8bd8662bcf2ef))
- ✅ **W&B run**: https://wandb.ai/atrey-dev/pharmarl
- ✅ **parse_rate=100%** confirmed (chat template fix is working)
- ✅ **HF Hub destination**: `anshumanatrey/pharmarl-llama-3b-trained-anshuman` (Anshuman's account)
- ❌ **NOT done**: H200 attempt (image incompatibility on first try, needs new stack)
- 📌 **Latest commit on main**: `52808d8` (chat template + strong SYSTEM + skip SFT)

The trainer file you'll be using is [`scripts/train_grpo.py`](scripts/train_grpo.py) — already has all fixes: chat template, bf16 LoRA cast, episode_id handling, error fallback, FAQ §17 W&B observability.

---

## Why H200 — research summary

| Component | H200 Compatible? | Source |
|---|---|---|
| **bitsandbytes 0.45.0+** | ✅ Full LLM.int8() for H100/H200/H800 | [BNB 0.45.0 release notes](https://github.com/bitsandbytes-foundation/bitsandbytes/releases/tag/0.45.0) |
| **`pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel`** | ✅ HF Jobs canonical CUDA-12.4 image | [HF Jobs guide](https://huggingface.co/docs/huggingface_hub/en/guides/jobs) |
| **xformers 0.0.30+** | ✅ Recent versions support sm_90 | [xformers PyPI](https://pypi.org/project/xformers/) |
| **CUDA 12.4** | ✅ Standard cu124 supports H200 | [Hopper Compatibility Guide](https://docs.nvidia.com/cuda/hopper-compatibility-guide/) |
| **xformers 0.0.27.post2** (our old pin) | ❌ Built for torch 2.4 only | — |
| **Multi-GPU H200** | ⚠️ Known CUDA error 802 — but we are SINGLE H200 so OK | [HF forum](https://discuss.huggingface.co/t/cuda-error-802-on-every-h200-multi-gpu-hf-job-across-three-vllm-images/175419) |

**Confidence H200 works first try with the right stack: 60-70%.** Failure modes are version-pin incompatibilities — same kind of thing we hit earlier. Each iteration costs ~30 min + ~$2.50.

---

## Setup checklist (you only need this once)

### 1. Install/update HF CLI

\`\`\`bash
pip install -U huggingface_hub
hf --version # should print 1.12.0 or newer
\`\`\`

### 2. Login with a write-scoped HF token

Get one at https://huggingface.co/settings/tokens (select \"Write\" permission).

\`\`\`bash
hf auth login
# Paste your token when prompted

hf auth whoami # should print YOUR HF username
\`\`\`

### 3. Get a WANDB API key (free)

Sign up at https://wandb.ai (free), then grab key from https://wandb.ai/authorize. We'll inject it as a job secret.

### 4. Verify HF credit on your account

Check via HF web UI → click avatar → \"Billing\" tab. Or fire a CPU sanity job (free) to confirm:

\`\`\`bash
hf jobs run python:3.12 python -c \"print('hi')\"
\`\`\`

### 5. List available hardware

\`\`\`bash
hf jobs hardware
\`\`\`

Confirm `h200` is listed. Pricing is `$5/hr`.

---

## Fire the H200 training run

Replace `YOUR_USERNAME` and `YOUR_WANDB_KEY` below.

\`\`\`bash
WANDB_API_KEY=\"YOUR_WANDB_KEY\" hf jobs run \\
--flavor h200 \\
--secrets HF_TOKEN \\
--secrets WANDB_API_KEY=\"\$WANDB_API_KEY\" \\
--timeout 4h \\
-d \\
pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel \\
bash -c '
set -e
echo \"=== H200 PHASE 2 — Vijay parallel attempt ===\"
apt-get update -qq && apt-get install -y -qq git
git clone https://github.com/AnshumanAtrey/pharmarl /pharmarl
cd /pharmarl
echo \"=== installing deps (NEW stack: torch 2.6 + xformers 0.0.30 + bnb 0.45.5) ===\"
pip install --no-cache-dir --progress-bar off \\
\"transformers==4.46.3\" \\
\"trl==0.13.0\" \\
\"unsloth==2025.2.15\" \\
\"unsloth_zoo==2025.2.7\" \\
\"peft>=0.12,<0.14\" \\
\"accelerate>=0.34,<1.0\" \\
\"bitsandbytes>=0.45.5,<0.46\" \\
\"xformers>=0.0.30,<0.1\" \\
\"wandb>=0.16,<1\" \\
\"datasets>=2.19,<3\" \\
\"huggingface_hub>=0.25,<1\" \\
\"rich\" \"typer\" \"requests\"
echo \"=== sanity: GPU detected? ===\"
python -c \"import torch; assert torch.cuda.is_available(); print(\\\"GPU:\\\", torch.cuda.get_device_name(0)); print(\\\"capability:\\\", torch.cuda.get_device_capability(0))\"
python -c \"from unsloth import FastLanguageModel; print(\\\"unsloth import OK\\\")\"
echo \"=== firing 200 GRPO steps on H200 ===\"
python -m scripts.train_grpo \\
--env-url https://anshumanatrey-pharmarl.hf.space \\
--model unsloth/Llama-3.2-3B-Instruct \\
--max-steps 200 \\
--sft-warmup-steps 0 \\
--num-generations 8 \\
--save-every 25 \\
--audit-every 25 \\
--hf-repo YOUR_USERNAME/pharmarl-llama-3b-trained-vijay-h200 \\
--hf-token \"\$HF_TOKEN\"
echo \"=== H200 training complete ===\"
'
\`\`\`

**What this command does:**
- `--flavor h200` → 1× H200 (141 GB, $5/h)
- `--secrets HF_TOKEN` → uses your locally-logged-in token
- `--secrets WANDB_API_KEY=...` → injects W&B key
- `--timeout 4h` → max 4h wall, well under H200's 1.7h expected runtime
- `-d` → detach so you can close the terminal
- New image `pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel` (cu124, supports H200)
- Bumped `xformers>=0.0.30` (older 0.0.27 doesn't support sm_90)
- Bumped `bitsandbytes>=0.45.5` (Hopper supported per release notes)
- Pushes the trained adapter to **YOUR account** (different repo than Anshuman's, no collision)

**Expected wall time on H200**: ~1.7 hours. **Cost**: ~$8.50.

---

## Watch the job

\`\`\`bash
hf jobs ps # find job ID
hf jobs logs --follow # tail logs
hf jobs inspect # check status
hf jobs cancel # if you need to abort
\`\`\`

**Success signals to watch (in order):**

1. `=== H200 PHASE 2 — Vijay parallel attempt ===` — bash booted ✅
2. `Successfully installed transformers-4.46.3 trl-0.13.0 ...` — pip done (~3-4 min)
3. `GPU: NVIDIA H200` and `capability: (9, 0)` — H200 detected ✅
4. `unsloth import OK` — unsloth+xformers compatible ✅
5. `[grpo] step= 0 ... parse=100%` — first training step (~5 min after boot)
6. `[grpo] step= 25 ... parse=100%` — every 25 steps after
7. `[push] done — https://huggingface.co/YOUR_USERNAME/pharmarl-llama-3b-trained-vijay-h200` — DONE ✅

**If `CUDA: False` or AssertionError at sanity step** → image still doesn't match H200. Cancel + try Plan B.

**If `Traceback` mid-training** → paste traceback in this issue, we'll iterate.

---

## Plan B — fallback if H200 image is incompatible

Fall back to `a100-large` with the OLD known-working image. Same flow, just different flavor + image:

\`\`\`bash
hf jobs run \\
--flavor a100-large \\
--secrets HF_TOKEN \\
--secrets WANDB_API_KEY=\"\$WANDB_API_KEY\" \\
--timeout 4h \\
-d \\
pytorch/pytorch:2.4.0-cuda12.1-cudnn9-devel \\
bash -c '
set -e
apt-get update -qq && apt-get install -y -qq git
git clone https://github.com/AnshumanAtrey/pharmarl /pharmarl
cd /pharmarl
pip install --no-cache-dir --progress-bar off \\
\"transformers==4.46.3\" \"trl==0.13.0\" \\
\"unsloth==2025.2.15\" \"unsloth_zoo==2025.2.7\" \\
\"peft>=0.12,<0.14\" \"accelerate>=0.34,<1.0\" \\
\"bitsandbytes>=0.44,<0.46\" \"xformers==0.0.27.post2\" \\
\"wandb>=0.16,<1\" \"datasets>=2.19,<3\" \\
\"huggingface_hub>=0.25,<1\" \"rich\" \"typer\" \"requests\"
python -m scripts.train_grpo \\
--env-url https://anshumanatrey-pharmarl.hf.space \\
--model unsloth/Llama-3.2-3B-Instruct \\
--max-steps 200 --sft-warmup-steps 0 --num-generations 8 \\
--save-every 25 --audit-every 25 \\
--hf-repo YOUR_USERNAME/pharmarl-llama-3b-trained-vijay-a100 \\
--hf-token \"\$HF_TOKEN\"
'
\`\`\`

A100 = 2× faster than a10g, queues unpredictably (we saw 60+ min waits today, sometimes <5 min). Expected wall ~2.8h.

---

## What happens after training succeeds

- Trained adapter lives at `https://huggingface.co/YOUR_USERNAME/pharmarl-llama-3b-trained-vijay-h200`
- Drop the URL in this issue so we can pull for evals
- We'll run held-out JNK3 eval against your adapter + Anshuman's, pick the stronger one

---

## Reference files in this repo

- [`scripts/train_grpo.py`](scripts/train_grpo.py) — the trainer (latest commit `52808d8` has all fixes)
- [`server/app.py`](server/app.py) — env served at `https://anshumanatrey-pharmarl.hf.space`
- [`models.py`](models.py) — observation/action schemas
- [`docs/training-fallback.md`](docs/training-fallback.md) — original Path C runbook
- [`docs/pitch-script.md`](docs/pitch-script.md) — pitch with `[FILL]` placeholders for trained-model numbers

## What we have already tried

- ✅ a10g-large with old pinned stack (parse=0%) → fixed with chat template
- ✅ a10g-large with chat template + sft=0 (parse=100%) → currently running v6
- ❌ h200 with `pytorch:2.4.0-cuda12.1` image → `CUDA: False` (image too old for Hopper)
- ❌ a100-large → queue lottery (sat 3+ min, gave up to fall back to a10g)

## Cost projection

- H200 attempt: ~$8.50 (best case, single iteration)
- 2-3 iterations: ~$15-20
- Your $30 credit → plenty of room. Anshuman's $30 stays on a10g.

## References

- [Bitsandbytes H100/H200 support](https://github.com/bitsandbytes-foundation/bitsandbytes/releases/tag/0.45.0)
- [HF Jobs official docs](https://huggingface.co/docs/huggingface_hub/en/guides/jobs)
- [Unsloth requirements](https://unsloth.ai/docs/get-started/fine-tuning-for-beginners/unsloth-requirements)
- [HF forum H200 multi-GPU known issue](https://discuss.huggingface.co/t/cuda-error-802-on-every-h200-multi-gpu-hf-job-across-three-vllm-images/175419)
- [NVIDIA Hopper Compatibility Guide](https://docs.nvidia.com/cuda/hopper-compatibility-guide/)

---

**Ping in this thread when you fire it. Anshuman is keeping the a10g run healthy as the safety net.**