THUDM / THUDM/slime

【模型支持问题】关于不同qwen 30BA3B模型支持的问题

Open
#1,340 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.5k
Forks
1.3k
Avg merge
5h 36m
Merged PRs (30d)
22

Description

  1. 请问slime/scripts/models/qwen3-30B-A3B.sh 脚本所支持的模型是这个吗:https://huggingface.co/Qwen/Qwen3-30B-A3B 。我在将这个模型转换成mcore后运行,发现报错,可能的原因是什么呢:
    Traceback (most recent call last):
    File "/home/jovyan/work/lyl/code/CodeReview/slime/train.py", line 106, in
    train(args)
    File "/home/jovyan/work/lyl/code/CodeReview/slime/train.py", line 27, in train
    actor_model, critic_model = create_training_models(args, pgs, rollout_manager)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/jovyan/work/lyl/code/CodeReview/slime/slime/ray/placement_group.py", line 143, in create_training_models
    start_rollout_ids = ray.get(
    ^^^^^^^^
    File "/usr/local/lib/python3.12/dist-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
    return fn(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.12/dist-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 2972, in get
    values, debugger_breakpoint = worker.get_objects(
    ^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 1031, in get_objects
    raise value.as_instanceof_cause()
    ray.exceptions.RayTaskError(CheckpointException): ray::MegatronTrainRayActor.init() (pid=3035762, ip=10.72.1.39, actor_id=f1edc61a1301fa5f41971cc702000000, repr=<slime.backends.megatron_utils.actor.MegatronTrainRayActor object at 0x763a5c898440>)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/jovyan/work/lyl/code/CodeReview/slime/slime/utils/context_utils.py", line 9, in wrapper
    return fn(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^^^
    File "/home/jovyan/work/lyl/code/CodeReview/slime/slime/backends/megatron_utils/actor.py", line 91, in init
    (self.model, self.optimizer, self.opt_param_scheduler, loaded_rollout_id) = initialize_model_and_optimizer(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/jovyan/work/lyl/code/CodeReview/slime/slime/backends/megatron_utils/model.py", line 726, in initialize_model_and_optimizer
    iteration, _ = load_checkpoint(
    ^^^^^^^^^^^^^^^^
    File "/home/jovyan/work/lyl/code/CodeReview/slime/slime/backends/megatron_utils/checkpoint.py", line 28, in load_checkpoint
    return _load_checkpoint_megatron(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/root/Megatron-LM/megatron/training/checkpointing.py", line 1544, in load_checkpoint
    state_dict, checkpoint_name, release, ckpt_type = _load_base_checkpoint(
    ^^^^^^^^^^^^^^^^^^^^^^
    File "/root/Megatron-LM/megatron/training/checkpointing.py", line 1090, in _load_base_checkpoint
    return _load_global_dist_base_checkpoint(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/root/Megatron-LM/megatron/training/checkpointing.py", line 984, in _load_global_dist_base_checkpoint
    state_dict = dist_checkpointing.load(sharded_state_dict, checkpoint_name, load_strategy, strict=args.dist_ckpt_strictness)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/root/Megatron-LM/megatron/core/dist_checkpointing/serialization.py", line 161, in load
    loaded_state_dict = sharded_strategy.load(sharded_state_dict, checkpoint_dir)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/root/Megatron-LM/megatron/core/dist_checkpointing/strategies/torch.py", line 925, in load
    checkpoint.load_state_dict(
    File "/usr/local/lib/python3.12/dist-packages/typing_extensions.py", line 3004, in wrapper
    return arg(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.12/dist-packages/torch/distributed/checkpoint/state_dict_loader.py", line 48, in load_state_dict
    return _load_state_dict(
    ^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.12/dist-packages/torch/distributed/checkpoint/state_dict_loader.py", line 279, in _load_state_dict
    central_plan = distW.reduce_scatter("plan", local_step, global_step)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.12/dist-packages/torch/distributed/checkpoint/utils.py", line 219, in reduce_scatter
    raise result
    torch.distributed.checkpoint.api.CheckpointException: CheckpointException ranks:dict_keys([0, 1, 2, 3, 4, 5, 6, 7])
    ray::MegatronTrainRayActor.init() (pid=3035762, ip=10.72.1.39, actor_id=f1edc61a1301fa5f41971cc702000000, repr=<slime.backends.megatron_utils.actor.MegatronTrainRayActor object at 0x763a5c898440>)
    File "/usr/local/lib/python3.12/dist-packages/torch/distributed/checkpoint/utils.py", line 192, in reduce_scatter
    local_data = map_fun()
    ^^^^^^^^^
    File "/usr/local/lib/python3.12/dist-packages/torch/distributed/checkpoint/logger.py", line 87, in wrapper
    result = func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.12/dist-packages/torch/distributed/checkpoint/state_dict_loader.py", line 266, in local_step
    local_plan = planner.create_local_plan()
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/root/Megatron-LM/megatron/core/dist_checkpointing/strategies/torch.py", line 616, in create_local_plan
    self._validate_global_shapes(self.metadata, self.shapes_validation_sharded_tensors)
    File "/root/Megatron-LM/megatron/core/dist_checkpointing/strategies/torch.py", line 569, in _validate_global_shapes
    raise KeyError(
    KeyError: "decoder.layers.0.self_attention.linear_proj.weight from model not in state dict: ['decoder.final_layernorm._extra_state/shard_0_1', 'decoder.final_layernorm.weight', 'decoder.layers.mlp.experts.experts.linear_fc1._extra_state/shard_0.0_48.128', 'decoder.layers.mlp.experts.experts.linear_fc1._extra_state/shard_0.100_48.128', 'decoder.layers.mlp.experts.experts.linear_fc1._extra_state/shard_0.101_48.128', 'decoder.layers.mlp.experts.experts.linear_fc1._extra_state/shard_0.102_48.128', 'decoder.layers.mlp.experts.experts.linear_fc1._extra_state/shard

  2. 如果希望跑 qwen 3 30BA3B-Instruct (https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507) ,应该怎么设置模型参数文件呢?是可以复用的吗?

  3. --loss-mask-type qwen3 这个参数是否有用?对不同的qwen3系列模型(instruct,thinking系列等),应该怎么设置这个参数?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with slime/scripts/models/qwen3-30B-A3B.sh and trace the reported failure through train.py, slime/ray/placement_group.py, slime/backends/megatron_utils/actor.py, model.py, and checkpoint.py. Compare the model's checkpoint keys with the expected keys, then verify how the script and --loss-mask-type qwen3 apply to the listed Qwen3 variants. Done means establishing supported models and documenting the required parameter settings or incompatibility.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.