【模型支持问题】关于不同qwen 30BA3B模型支持的问题
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.5k
- Forks
- 1.3k
- Avg merge
- 5h 36m
- Merged PRs (30d)
- 22
Description
-
请问slime/scripts/models/qwen3-30B-A3B.sh 脚本所支持的模型是这个吗:https://huggingface.co/Qwen/Qwen3-30B-A3B 。我在将这个模型转换成mcore后运行,发现报错,可能的原因是什么呢:
Traceback (most recent call last):
File "/home/jovyan/work/lyl/code/CodeReview/slime/train.py", line 106, in
train(args)
File "/home/jovyan/work/lyl/code/CodeReview/slime/train.py", line 27, in train
actor_model, critic_model = create_training_models(args, pgs, rollout_manager)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jovyan/work/lyl/code/CodeReview/slime/slime/ray/placement_group.py", line 143, in create_training_models
start_rollout_ids = ray.get(
^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 2972, in get
values, debugger_breakpoint = worker.get_objects(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 1031, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(CheckpointException): ray::MegatronTrainRayActor.init() (pid=3035762, ip=10.72.1.39, actor_id=f1edc61a1301fa5f41971cc702000000, repr=<slime.backends.megatron_utils.actor.MegatronTrainRayActor object at 0x763a5c898440>)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jovyan/work/lyl/code/CodeReview/slime/slime/utils/context_utils.py", line 9, in wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/jovyan/work/lyl/code/CodeReview/slime/slime/backends/megatron_utils/actor.py", line 91, in init
(self.model, self.optimizer, self.opt_param_scheduler, loaded_rollout_id) = initialize_model_and_optimizer(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jovyan/work/lyl/code/CodeReview/slime/slime/backends/megatron_utils/model.py", line 726, in initialize_model_and_optimizer
iteration, _ = load_checkpoint(
^^^^^^^^^^^^^^^^
File "/home/jovyan/work/lyl/code/CodeReview/slime/slime/backends/megatron_utils/checkpoint.py", line 28, in load_checkpoint
return _load_checkpoint_megatron(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/Megatron-LM/megatron/training/checkpointing.py", line 1544, in load_checkpoint
state_dict, checkpoint_name, release, ckpt_type = _load_base_checkpoint(
^^^^^^^^^^^^^^^^^^^^^^
File "/root/Megatron-LM/megatron/training/checkpointing.py", line 1090, in _load_base_checkpoint
return _load_global_dist_base_checkpoint(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/Megatron-LM/megatron/training/checkpointing.py", line 984, in _load_global_dist_base_checkpoint
state_dict = dist_checkpointing.load(sharded_state_dict, checkpoint_name, load_strategy, strict=args.dist_ckpt_strictness)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/Megatron-LM/megatron/core/dist_checkpointing/serialization.py", line 161, in load
loaded_state_dict = sharded_strategy.load(sharded_state_dict, checkpoint_dir)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/Megatron-LM/megatron/core/dist_checkpointing/strategies/torch.py", line 925, in load
checkpoint.load_state_dict(
File "/usr/local/lib/python3.12/dist-packages/typing_extensions.py", line 3004, in wrapper
return arg(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/checkpoint/state_dict_loader.py", line 48, in load_state_dict
return _load_state_dict(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/checkpoint/state_dict_loader.py", line 279, in _load_state_dict
central_plan = distW.reduce_scatter("plan", local_step, global_step)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/checkpoint/utils.py", line 219, in reduce_scatter
raise result
torch.distributed.checkpoint.api.CheckpointException: CheckpointException ranks:dict_keys([0, 1, 2, 3, 4, 5, 6, 7])
ray::MegatronTrainRayActor.init() (pid=3035762, ip=10.72.1.39, actor_id=f1edc61a1301fa5f41971cc702000000, repr=<slime.backends.megatron_utils.actor.MegatronTrainRayActor object at 0x763a5c898440>)
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/checkpoint/utils.py", line 192, in reduce_scatter
local_data = map_fun()
^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/checkpoint/logger.py", line 87, in wrapper
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/checkpoint/state_dict_loader.py", line 266, in local_step
local_plan = planner.create_local_plan()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/Megatron-LM/megatron/core/dist_checkpointing/strategies/torch.py", line 616, in create_local_plan
self._validate_global_shapes(self.metadata, self.shapes_validation_sharded_tensors)
File "/root/Megatron-LM/megatron/core/dist_checkpointing/strategies/torch.py", line 569, in _validate_global_shapes
raise KeyError(
KeyError: "decoder.layers.0.self_attention.linear_proj.weight from model not in state dict: ['decoder.final_layernorm._extra_state/shard_0_1', 'decoder.final_layernorm.weight', 'decoder.layers.mlp.experts.experts.linear_fc1._extra_state/shard_0.0_48.128', 'decoder.layers.mlp.experts.experts.linear_fc1._extra_state/shard_0.100_48.128', 'decoder.layers.mlp.experts.experts.linear_fc1._extra_state/shard_0.101_48.128', 'decoder.layers.mlp.experts.experts.linear_fc1._extra_state/shard_0.102_48.128', 'decoder.layers.mlp.experts.experts.linear_fc1._extra_state/shard -
如果希望跑 qwen 3 30BA3B-Instruct (https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507) ,应该怎么设置模型参数文件呢?是可以复用的吗?
-
--loss-mask-type qwen3 这个参数是否有用?对不同的qwen3系列模型(instruct,thinking系列等),应该怎么设置这个参数?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with slime/scripts/models/qwen3-30B-A3B.sh and trace the reported failure through train.py, slime/ray/placement_group.py, slime/backends/megatron_utils/actor.py, model.py, and checkpoint.py. Compare the model's checkpoint keys with the expected keys, then verify how the script and --loss-mask-type qwen3 apply to the listed Qwen3 variants. Done means establishing supported models and documenting the required parameter settings or incompatibility.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100