CUDA Error: invalid argument at /workspace/csrc/cumem_allocator.cpp:119

Checklist / 检查清单

• I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues，确认这是一个新的 bug report。

Bug Description / Bug 描述

我使用如下训练命令启动qwen3-vl-30b-a3b的moe模型grpo训练，使用msswift官方制作的镜像modelscope-registry.cn-hangzhou.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.8.1-py311-torch2.10.0-vllm0.17.1-modelscope1.34.0-swift4.0.3：

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
NPROC_PER_NODE=8
MASTER_ADDR=127.0.0.1
MASTER_PORT=29500
NNODES=1
NODE_RANK=0
IMAGE_MAX_TOKEN_NUM=1024
OMP_NUM_THREADS=14
megatron rlhf
--rlhf_type grpo
--mcore_model /tmp/Qwen3-VL-30B-A3B-Instruct-mcore
--dataset /tmp/grpo_repro.jsonl
--split_dataset_ratio 0.0
--reward_funcs format
--moe_permute_fusion true
--context_parallel_size 1
--tensor_model_parallel_size 4
--sequence_parallel true
--expert_tensor_parallel_size 1
--expert_model_parallel_size 8
--pipeline_model_parallel_size 1
--moe_grouped_gemm true
--moe_shared_expert_overlap true
--moe_aux_loss_coeff 1e-4
--micro_batch_size 1
--global_batch_size 64
--steps_per_generation 1
--num_generations 8
--use_vllm true
--vllm_mode colocate
--vllm_gpu_memory_utilization 0.3
--vllm_tensor_parallel_size 4
--vllm_max_model_len 26000
--tuner_type lora
--lora_rank 16
--lora_alpha 32
--target_modules linear_qkv linear_proj
--bf16 true
--num_train_epochs 1
--finetune true
--freeze_vit true
--freeze_aligner true
--vit_gradient_checkpointing false
--recompute_granularity full
--recompute_method uniform
--recompute_num_layers 1
--lr 2e-5
--min_lr 1e-6
--lr_warmup_fraction 0.05
--max_length 24000
--max_completion_length 1536
--temperature 1.0
--top_p 0.9
--beta 0.001
--importance_sampling_level sequence
--epsilon 0.2
--epsilon_high 0.28
--dynamic_sample true
--max_resample_times 3
--overlong_filter true
--loss_type grpo
--sleep_level 2
--offload_model true
--offload_bridge true
--offload_optimizer true
--optimizer_cpu_offload true
--use_precision_aware_optimizer true
--optimizer_offload_fraction 1
--eval_steps 250
--save_steps 250
--logging_steps 5
--output_dir /tmp/grpo_qwen3vl_moe_colocate_repro
--dataset_num_proc 1
--dataloader_num_workers 1
--attention_backend flash
--padding_free true
--log_completions true
--gradient_accumulation_fusion false
--no_save_optim true
--no_save_rng true

遇到了如下error，核心应该是invalid argument at cumem_allocator.cpp:119（因为只有几十 MiB 申请失败，但 GPU 明明还有大量空闲显存）：

CUDA Error: invalid argument at /workspace/csrc/cumem_allocator.cpp:119
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/swift/cli/_megatron/rlhf.py", line 7, in
megatron_rlhf_main()
File "/usr/local/lib/python3.11/site-packages/swift/megatron/pipelines/train/rlhf.py", line 73, in megatron_rlhf_main
return MegatronRLHF(args).main()
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/swift/pipelines/base.py", line 52, in main
result = self.run()
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/swift/megatron/pipelines/train/sft.py", line 66, in run
trainer = self.prepare_trainer()
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/swift/megatron/pipelines/train/rlhf.py", line 34, in prepare_trainer
return trainer_cls(args, self.template, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/swift/megatron/trainers/grpo_trainer.py", line 54, in init
self._init_rollout_engine()
File "/usr/local/lib/python3.11/site-packages/swift/megatron/trainers/grpo_trainer.py", line 122, in _init_rollout_engine
super()._init_rollout_engine()
File "/usr/local/lib/python3.11/site-packages/swift/megatron/trainers/rollout_mixin.py", line 222, in _init_rollout_engine
self.engine = self._prepare_vllm_engine()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/swift/megatron/trainers/rollout_mixin.py", line 248, in _prepare_vllm_engine
engine = GRPOVllmEngine(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/swift/infer_engine/vllm_engine.py", line 161, in init
self._prepare_engine()
File "/usr/local/lib/python3.11/site-packages/swift/infer_engine/vllm_engine.py", line 183, in _prepare_engine
engine = llm_engine_cls.from_engine_args(self.engine_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/vllm/v1/engine/llm_engine.py", line 177, in from_engine_args
return cls(
^^^^
File "/usr/local/lib/python3.11/site-packages/vllm/v1/engine/llm_engine.py", line 111, in init
self.engine_core = EngineCoreClient.make_client(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/vllm/v1/engine/core_client.py", line 100, in make_client
return InprocClient(vllm_config, executor_class, log_stats)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/vllm/v1/engine/core_client.py", line 282, in init
self.engine_core = EngineCore(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 110, in init
self.model_executor = executor_class(vllm_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/vllm/v1/executor/abstract.py", line 103, in init
self._init_executor()
File "/usr/local/lib/python3.11/site-packages/vllm/v1/executor/uniproc_executor.py", line 154, in _init_executor
super()._init_executor()
File "/usr/local/lib/python3.11/site-packages/vllm/v1/executor/uniproc_executor.py", line 49, in _init_executor
self.driver_worker.load_model()
File "/usr/local/lib/python3.11/site-packages/vllm/v1/worker/gpu_worker.py", line 337, in load_model
self.model_runner.load_model(load_dummy_weights=dummy_weights)
File "/usr/local/lib/python3.11/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4363, in load_model
raise e
File "/usr/local/lib/python3.11/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4297, in load_model
self.model = model_loader.load_model(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/vllm/model_executor/model_loader/base_loader.py", line 54, in load_model
model = initialize_model(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/vllm/model_executor/model_loader/utils.py", line 56, in initialize_model
model = model_class(vllm_config=vllm_config, prefix=prefix)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/vllm/model_executor/models/qwen3_vl_moe.py", line 437, in init
self.visual = Qwen3_VisionTransformer(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/vllm/model_executor/models/qwen3_vl.py", line 355, in init
self.patch_embed = Qwen3_VisionPatchEmbed(
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/vllm/model_executor/models/qwen3_vl.py", line 161, in init
self.proj = Conv3dLayer(
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/vllm/model_executor/layers/conv.py", line 85, in init
torch.empty(
File "/usr/local/lib/python3.11/site-packages/torch/utils/_device.py", line 109, in torch_function
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 7 has a total capacity of 79.11 GiB of which 66.43 GiB is free. Including non-PyTorch memory, this process has 0 bytes memory in use. Of the allocated memory 8.51 GiB is allocated by PyTorch, and 856.16 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

我按照CUDA Error: invalid argument at /workspace/csrc/cumem_allocator.cpp:119进行了检索，有人遇到了类似的问题：
https://github.com/vllm-project/vllm/issues/35463
其解决方案是：
https://github.com/vllm-project/vllm/pull/35489

这个issue里提到可以尝试换成server模式而不是colocate模式，或者用更高版本的vllm（在当前镜像下，我直接升级vllm版本是可行的吗？）https://github.com/modelscope/ms-swift/issues/9144

我已经把micro batch size设置为了1，依旧有一样的报错，所以我认为不是真的发生了OOM，而是vllm申请内存或者是什么阶段发生了问题

How to Reproduce / 如何复现

见前面的bug描述

Additional Information / 补充信息

No response

Start with the provided reproduction and the traceback through swift/megatron/trainers/rollout_mixin.py, swift/infer_engine/vllm_engine.py, and vllm/model_executor/models/qwen3_vl.py, especially Conv3dLayer initialization. Compare the colocate configuration and vLLM versions with vLLM issue 35463 and pull request 35489. Done means the same Qwen3-VL GRPO startup completes without the invalid-argument or misleading out-of-memory failure.