modelscope / modelscope/ms-swift
[Bug]: Megatron arg mtp_shared_weights is defined but never consumed (no effect)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
[Bug]: Megatron arg mtp_shared_weights is defined but never consumed (no effect)
Describe the bug
mtp_shared_weights is declared as a Megatron training argument but is never read anywhere in the codebase, so setting --mtp_shared_weights true has no effect. Users expecting it to make the N MTP layers share one physical layer (a single repeated MTP layer, as some inference stacks deploy) will silently get N independent layers instead.
Verification
In both 4.4.2 and the latest 4.5.2, the only occurrence of the symbol in the whole package is its definition:
swift/megatron/arguments/megatron_args.py: mtp_shared_weights: bool = False
# reproduce
pip download ms-swift==4.5.2 --no-deps --no-binary :all: -d /tmp/sw
cd /tmp/sw && tar xzf ms_swift-4.5.2.tar.gz
grep -rn "mtp_shared_weights" ms_swift-4.5.2/swift
# -> only the single definition line; no consumer, no plumbing into the mcore MTP block
There is no code path passing this flag into get_gpt_mtp_block_spec / the Megatron MultiTokenPredictionBlock, and no branch that builds a single shared layer when it is True.
Expected behavior
Either:
- Wire
mtp_shared_weightsthrough to the mcore MTP block so a single physical MTP layer is built and reused across depths (mcore ≥ 0.19 exposesmtp_use_repeated_layerinTransformerConfigfor exactly this), or - Remove the dead argument to avoid the silent no-op.
Context / motivation
Speculative-decoding deployment for MTP often runs layer0 autoregressively K times (one physical draft layer, applied K times). Training with a single repeated MTP layer matches that deployment structure exactly, whereas N independent layers do not — the extra layers are never executed by the serving engine. A working mtp_shared_weights would let users train the deployment-aligned single-layer drafter directly.
Environment
- ms-swift 4.4.2 and 4.5.2
- megatron backend (mcore-bridge)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with swift/megatron/arguments/megatron_args.py and trace the Megatron path around get_gpt_mtp_block_spec and MultiTokenPredictionBlock. Verify how TransformerConfig is built and whether mcore exposes mtp_use_repeated_layer. Done means the flag has an effective consumer that creates shared MTP weights, or the unused argument is removed, with the grep check showing no silent definition-only behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100