togethercomputer / togethercomputer/xorl
sync_inference_weights corrupts the sampler for dense Qwen3 models with tied embeddings
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 46
- Forks
- 1
- Avg merge
- 4h 16m
- Merged PRs (30d)
- 4
Description
Minimal repro (dense Qwen3-4B-Instruct-2507, tie_word_embeddings: true):
- Launch the trainer in full-weight mode on the exact dense-Qwen3 contract (
ce_mode: bi_fused, fa4,merge_qkv: false) and xorl-sglang with--rl-on-policy-target xorl --enable-fp32-lm-headon the same model. - Fresh sampler generates perfectly (
<guess>CRANE</guess>for the Wordle prompt). add_inference_endpoint+sync_weights_to_inference(sync_method='nccl_broadcast')from the untrained trainer — i.e. byte-identical weights. Sync reports success: Synced 399 params to 1 endpoint(s), 8,822,848,512 bytes.- The same prompt now yields deterministic garbage (
'arn讹tereg游客ashingtonoins Prompt trứng…'— identical string across repeated syncs/restarts, so the sync deterministically writes wrong bytes).
Likely suspect: tied embeddings. Qwen3-4B has tie_word_embeddings: true (Qwen3-8B does not); a scrambled lm_head/embedding readout produces exactly this symptom (coherent-frequency random tokens from step one). Possibly interacting with the exact-contract trainer's wrapped/Class-B parameter representations at extraction time.
Impact: blocks the endpoint-driven Wordle zero-K3 recipe (xorl-client examples/wordle) on tied-embedding dense models via full-weight sync. Found while reproducing the train-inference-mismatch write-up's Wordle experiment (see #84 thread for the K₃ measurements that motivated the full-weight route).
Env: 2×H100, torch 2.11, branch qywu/value-head-critic (sync path identical to main).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with add_inference_endpoint and sync_weights_to_inference(sync_method='nccl_broadcast'), using the dense Qwen3-4B-Instruct-2507 contract and the Wordle reproduction described here. Trace parameter extraction and synchronization for tied embeddings, then verify that syncing byte-identical untrained weights preserves the sampler output and no longer produces deterministic garbage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100