togethercomputer / togethercomputer/xorl

sync_inference_weights corrupts the sampler for dense Qwen3 models with tied embeddings

Open
#87 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
46
Forks
1
Avg merge
4h 16m
Merged PRs (30d)
4

Description

Minimal repro (dense Qwen3-4B-Instruct-2507, tie_word_embeddings: true):

  1. Launch the trainer in full-weight mode on the exact dense-Qwen3 contract (ce_mode: bi_fused, fa4, merge_qkv: false) and xorl-sglang with --rl-on-policy-target xorl --enable-fp32-lm-head on the same model.
  2. Fresh sampler generates perfectly (<guess>CRANE</guess> for the Wordle prompt).
  3. add_inference_endpoint + sync_weights_to_inference(sync_method='nccl_broadcast') from the untrained trainer — i.e. byte-identical weights. Sync reports success: Synced 399 params to 1 endpoint(s), 8,822,848,512 bytes.
  4. The same prompt now yields deterministic garbage ('arn讹tereg游客ashingtonoins Prompt trứng…' — identical string across repeated syncs/restarts, so the sync deterministically writes wrong bytes).

Likely suspect: tied embeddings. Qwen3-4B has tie_word_embeddings: true (Qwen3-8B does not); a scrambled lm_head/embedding readout produces exactly this symptom (coherent-frequency random tokens from step one). Possibly interacting with the exact-contract trainer's wrapped/Class-B parameter representations at extraction time.

Impact: blocks the endpoint-driven Wordle zero-K3 recipe (xorl-client examples/wordle) on tied-embedding dense models via full-weight sync. Found while reproducing the train-inference-mismatch write-up's Wordle experiment (see #84 thread for the K₃ measurements that motivated the full-weight route).

Env: 2×H100, torch 2.11, branch qywu/value-head-critic (sync path identical to main).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with add_inference_endpoint and sync_weights_to_inference(sync_method='nccl_broadcast'), using the dense Qwen3-4B-Instruct-2507 contract and the Wordle reproduction described here. Trace parameter extraction and synchronization for tied embeddings, then verify that syncing byte-identical untrained weights preserves the sampler output and no longer produces deterministic garbage.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.