modelscope / modelscope/ms-swift
rollout CUDA error: an illegal memory access was encountered
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Describe the bug
What the bug is, and how to reproduce, better with screenshots
I run server mode GRPO with customized multi-turn scheduler and --vllm_use_async_engine true
But CUDA error: An illegal memory access was encountered.
This error sometimes doesn't occur. I have searched some info and it might be a problem with vllm. But my machine is a bit special. If vllm is upgraded from 0.11.0 to 0.11.1 or a newer version, I need to reinstall all of torch, flash_attn, etc., which is very troublesome.
I'm not sure where the problem lies.
Additionally, my custom Scheduler includes multimodal images and tool calls. Different images need to be input for each turn(so I overwrite rollout_infos images[] in every turn). And after each turn of generation, there is a quality check step, and it is time-consuming, and the result of the quality check step will be added to the input for the next turn of generation. This is quite complex. I'm not sure if this CUDA error is related to these factors.
rollout log
[v1/executor/multiproc_executor.py:671] WorkerProc hit an exception.
[v1/executor/multiproc_executor.py:671] Traceback (most recent call last):
[v1/executor/multiproc_executor.py:671] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 666, in worker_busy_loop
[v1/executor/multiproc_executor.py:671] output = func(*args, **kwargs)
[v1/executor/multiproc_executor.py:671] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
[v1/executor/multiproc_executor.py:671] return func(*args, **kwargs)
[v1/executor/multiproc_executor.py:671] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 447, in execute_model
[v1/executor/multiproc_executor.py:671] output = self.model_runner.execute_model(scheduler_output,
[v1/executor/multiproc_executor.py:671] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
[v1/executor/multiproc_executor.py:671] return func(*args, **kwargs)
[v1/executor/multiproc_executor.py:671] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 2331, in execute_model
[v1/executor/multiproc_executor.py:671] logits = self.model.compute_logits(sample_hidden_states)
[v1/executor/multiproc_executor.py:671] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_vl.py", line 1594, in compute_logits
[v1/executor/multiproc_executor.py:671] return self.language_model.compute_logits(hidden_states)
[v1/executor/multiproc_executor.py:671] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/model_executor/models/qwen3.py", line 331, in compute_logits
[v1/executor/multiproc_executor.py:671] logits = self.logits_processor(self.lm_head, hidden_states)
[v1/executor/multiproc_executor.py:671] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
[v1/executor/multiproc_executor.py:671] return self._call_impl(*args, **kwargs)
[v1/executor/multiproc_executor.py:671] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
[v1/executor/multiproc_executor.py:671] return forward_call(*args, **kwargs)
[v1/executor/multiproc_executor.py:671] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/model_executor/layers/logits_processor.py", line 58, in forward
[v1/executor/multiproc_executor.py:671] logits = self._get_logits(hidden_states, lm_head, embedding_bias)
[v1/executor/multiproc_executor.py:671] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/model_executor/layers/logits_processor.py", line 95, in _get_logits
[v1/executor/multiproc_executor.py:671] logits = self._gather_logits(logits)
[v1/executor/multiproc_executor.py:671] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/model_executor/layers/logits_processor.py", line 77, in _gather_logits
[v1/executor/multiproc_executor.py:671] logits = tensor_model_parallel_all_gather(logits)
[v1/executor/multiproc_executor.py:671] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/distributed/communication_op.py", line 20, in tensor_model_parallel_all_gather
[v1/executor/multiproc_executor.py:671] return get_tp_group().all_gather(input_, dim)
[v1/executor/multiproc_executor.py:671] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 378, in all_gather
[v1/executor/multiproc_executor.py:671] return torch.ops.vllm.all_gather(input_,
[v1/executor/multiproc_executor.py:671] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/torch/_ops.py", line 1243, in __call__
[v1/executor/multiproc_executor.py:671] return self._op(*args, **kwargs)
[v1/executor/multiproc_executor.py:671] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 141, in all_gather
[v1/executor/multiproc_executor.py:671] return group._all_gather_out_place(tensor, dim)
[v1/executor/multiproc_executor.py:671] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 389, in _all_gather_out_place
[v1/executor/multiproc_executor.py:671] return self.device_communicator.all_gather(input_, dim)
[v1/executor/multiproc_executor.py:671] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/distributed/device_communicators/base_device_communicator.py", line 143, in all_gather
[v1/executor/multiproc_executor.py:671] dist.all_gather_into_tensor(output_tensor,
[v1/executor/multiproc_executor.py:671] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
[v1/executor/multiproc_executor.py:671] return func(*args, **kwargs)
[v1/executor/multiproc_executor.py:671] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3986, in all_gather_into_tensor
[v1/executor/multiproc_executor.py:671] work = group._allgather_base(output_tensor, input_tensor, opts)
[v1/executor/multiproc_executor.py:671] torch.AcceleratorError: CUDA error: an illegal memory access was encountered
[v1/executor/multiproc_executor.py:671] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
[v1/executor/multiproc_executor.py:671] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
[v1/executor/multiproc_executor.py:671] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
[v1/executor/multiproc_executor.py:671]
[logging_utils/dump_input.py:69] Dumping input data for V1 LLM engine (v0.11.0) with config: model='/checkpoint/binary/train_package/output/models/Qwen3-VL-32B-Instruct/sft4grpo', speculative_config=None, tokenizer='/checkpoint/binary/train_package/output/models/Qwen3-VL-32B-Instruct/sft4grpo', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=65536, download_dir=None, load_format=dummy, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=2, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/checkpoint/binary/train_package/output/models/Qwen3-VL-32B-Instruct/sft4grpo, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2","vllm.mamba_mixer","vllm.short_conv","vllm.linear_attention","vllm.plamo2_mamba_mixer","vllm.gdn_attention","vllm.sparse_attn_indexer"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":[2,1],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"use_inductor_graph_partition":false,"pass_config":{},"max_capture_size":512,"local_cache_dir":null},
[logging_utils/dump_input.py:76] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=chatcmpl-d0bd4dff81b14373927f838342ed8f03,prompt_token_ids_len=4015,mm_features=[MultiModalFeatureSpec(data={'pixel_values': MultiModalFieldElem(modality='image', key='pixel_values', data=tensor([[-1.0000, -1.0000, -1.0000, ..., -1.0000, -1.0000, -1.0000],
[logging_utils/dump_input.py:76] [-1.0000, -1.0000, -1.0000, ..., -1.0000, -1.0000, -1.0000],
[logging_utils/dump_input.py:76] [-1.0000, -1.0000, -1.0000, ..., -1.0000, -1.0000, -1.0000],
[logging_utils/dump_input.py:76] ...,
[logging_utils/dump_input.py:76] [-1.0000, -1.0000, -1.0000, ..., -1.0000, -1.0000, -1.0000],
[logging_utils/dump_input.py:76] [-0.0197, -1.0000, -1.0000, ..., -1.0000, -1.0000, -1.0000],
[logging_utils/dump_input.py:76] [-1.0000, -1.0000, -1.0000, ..., -1.0000, -1.0000, -1.0000]],
[logging_utils/dump_input.py:76] dtype=torch.bfloat16), field=MultiModalFlatField(slices=[[slice(0, 1564, None)]], dim=0)), 'image_grid_thw': MultiModalFieldElem(modality='image', key='image_grid_thw', data=tensor([ 1, 34, 46]), field=MultiModalBatchedField())}, modality='image', identifier='b54e140f6a4ef3ac1f53d9f0729780bf1d4b4f54f875f202481140782c1e6eef', mm_position=PlaceholderRange(offset=2420, length=391, is_embed=None))],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.7, top_p=0.9, top_k=50, min_p=0.0, seed=None, stop=['<|endoftext|>', '<|im_end|>'], stop_token_ids=[151643, 151645], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=32768, min_tokens=0, logprobs=0, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, structured_outputs=None, extra_args=None),block_ids=([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 1782, 1790, 1798, 1806, 1814, 1822, 1830, 1838, 1846, 1857, 1865, 1873, 1881, 1889, 1897, 1905, 1913, 1921, 1929, 1937, 1945, 1953, 1961, 1969, 1977, 1985, 1993, 2001, 2009, 2017, 2025, 2033, 2041, 2049, 2057, 2065, 2073, 2081, 2089, 2097, 2105, 2113, 2121, 2129, 2137, 2145, 2153, 2161, 2169, 2177, 2185, 2358, 2359, 2360, 2361, 2362, 2363, 2364, 2365, 2366, 2367, 2368, 2369, 2370, 2371, 2372, 2373, 2374, 2375, 2376, 2377, 2378, 2379, 2380],),num_computed_tokens=3648,lora_request=None,prompt_embeds_shape=None)], scheduled_cached_reqs=CachedRequestData(req_ids=['chatcmpl-37eb06fce9fd4c26a31222e13ed67316', 'chatcmpl-6841bef0aadb4a01bf54ec5614891f82', 'chatcmpl-c56f7ddc436c4066bb4d70abb0eb8de9', 'chatcmpl-951907e69f3448f48a8d358bd7f10359', 'chatcmpl-3738bc6492f2459c9811393154a6e2cf', 'chatcmpl-f46415f77a5c478bb2ff39268784dc87', 'chatcmpl-b941b7397b0a453bbfbac9299e2e640c', 'chatcmpl-69725f9d73f74c9a88ae6109c30fb0e7'], resumed_from_preemption=[false, false, false, false, false, false, false, false], new_token_ids=[], new_block_ids=[null, null, null, null, null, null, null, null], num_computed_tokens=[3980, 3979, 3979, 3979, 27595, 27595, 27595, 27595]), num_scheduled_tokens={chatcmpl-69725f9d73f74c9a88ae6109c30fb0e7: 1, chatcmpl-b941b7397b0a453bbfbac9299e2e640c: 1, chatcmpl-951907e69f3448f48a8d358bd7f10359: 1, chatcmpl-d0bd4dff81b14373927f838342ed8f03: 367, chatcmpl-f46415f77a5c478bb2ff39268784dc87: 1, chatcmpl-c56f7ddc436c4066bb4d70abb0eb8de9: 1, chatcmpl-6841bef0aadb4a01bf54ec5614891f82: 1, chatcmpl-3738bc6492f2459c9811393154a6e2cf: 1, chatcmpl-37eb06fce9fd4c26a31222e13ed67316: 1}, total_num_scheduled_tokens=375, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[55], finished_req_ids=[], free_encoder_mm_hashes=[], structured_output_request_ids={}, grammar_bitmask=null, kv_connector_metadata=null)
[v1/engine/core.py:710] EngineCore encountered a fatal error.
[v1/engine/core.py:710] Traceback (most recent call last):
[v1/engine/core.py:710] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 701, in run_engine_core
[v1/engine/core.py:710] engine_core.run_busy_loop()
[v1/engine/core.py:710] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1045, in run_busy_loop
[v1/engine/core.py:710] executed = self._process_engine_step()
[v1/engine/core.py:710] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 754, in _process_engine_step
[v1/engine/core.py:710] outputs, model_executed = self.step_fn()
[v1/engine/core.py:710] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 284, in step
[v1/engine/core.py:710] model_output = self.execute_model_with_error_logging(
[v1/engine/core.py:710] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 270, in execute_model_with_error_logging
[v1/engine/core.py:710] raise err
[v1/engine/core.py:710] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 261, in execute_model_with_error_logging
[v1/engine/core.py:710] return model_fn(scheduler_output)
[v1/engine/core.py:710] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 181, in execute_model
[v1/engine/core.py:710] (output, ) = self.collective_rpc(
[v1/engine/core.py:710] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 264, in collective_rpc
[v1/engine/core.py:710] result = get_response(w, dequeue_timeout,
[v1/engine/core.py:710] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 248, in get_response
[v1/engine/core.py:710] raise RuntimeError(
[v1/engine/core.py:710] RuntimeError: Worker failed with error 'CUDA error: an illegal memory access was encountered
[v1/engine/core.py:710] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
[v1/engine/core.py:710] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
[v1/engine/core.py:710] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
[v1/engine/core.py:710] ', please check the stack trace above for the root cause
ERROR 12-03 18:53:09 [v1/engine/async_llm.py:480] AsyncLLM output_handler failed.
ERROR 12-03 18:53:09 [v1/engine/async_llm.py:480] Traceback (most recent call last):
ERROR 12-03 18:53:09 [v1/engine/async_llm.py:480] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 439, in output_handler
ERROR 12-03 18:53:09 [v1/engine/async_llm.py:480] outputs = await engine_core.get_output_async()
ERROR 12-03 18:53:09 [v1/engine/async_llm.py:480] File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 846, in get_output_async
ERROR 12-03 18:53:09 [v1/engine/async_llm.py:480] raise self._format_exception(outputs) from None
ERROR 12-03 18:53:09 [v1/engine/async_llm.py:480] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
[ERROR:swift] [D2C Turn 1] Error: EngineCore encountered an issue. See stack trace (above) for the root cause.
[v1/executor/multiproc_executor.py:558] Parent process exited, terminating worker
[v1/executor/multiproc_executor.py:599] WorkerProc shutting down.
Traceback (most recent call last):
File "/checkpoint/binary/train_package/train/plugin.py", line 760, in run
initial_output, current_similarity = await self._execute_initial_generation(
File "/checkpoint/binary/train_package/train/plugin.py", line 387, in _execute_initial_generation
response = await self.infer_engine.infer_async(initial_request, request_config, **kwargs)
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/swift/llm/infer/infer_engine/vllm_engine.py", line 790, in infer_async
return await self._infer_full_async(**kwargs)
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/swift/llm/infer/infer_engine/vllm_engine.py", line 655, in _infer_full_async
async for result in result_generator:
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 387, in generate
out = q.get_nowait() or await q.get()
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/v1/engine/output_processor.py", line 59, in get
raise output
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 439, in output_handler
outputs = await engine_core.get_output_async()
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 846, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
[v1/executor/multiproc_executor.py:558] Parent process exited, terminating worker
[v1/executor/multiproc_executor.py:558] Parent process exited, terminating worker
[ERROR:swift] [D2C Turn 1] Error: EngineCore encountered an issue. See stack trace (above) for the root cause.
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
[ERROR:swift] [D2C Turn 1] Error: EngineCore encountered an issue. See stack trace (above) for the root cause.
Traceback (most recent call last):
File "/checkpoint/binary/train_package/train/plugin.py", line 760, in run
initial_output, current_similarity = await self._execute_initial_generation(
File "/checkpoint/binary/train_package/train/plugin.py", line 387, in _execute_initial_generation
response = await self.infer_engine.infer_async(initial_request, request_config, **kwargs)
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/swift/llm/infer/infer_engine/vllm_engine.py", line 790, in infer_async
return await self._infer_full_async(**kwargs)
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/swift/llm/infer/infer_engine/vllm_engine.py", line 655, in _infer_full_async
async for result in result_generator:
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 387, in generate
out = q.get_nowait() or await q.get()
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/v1/engine/output_processor.py", line 59, in get
raise output
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 439, in output_handler
outputs = await engine_core.get_output_async()
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 846, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
[ERROR:swift] Method execution failed: async_infer
Traceback (most recent call last):
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/swift/llm/infer/rollout.py", line 283, in async_llm_worker
result = await method(*args, **kwargs)
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/swift/plugin/multi_turn.py", line 92, in async_infer
results = await self.infer_engine._batch_infer_stream(tasks, request_config.stream, use_tqdm, None)
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/swift/llm/infer/infer_engine/grpo_vllm_engine.py", line 180, in _batch_infer_stream
return await self.batch_run(new_tasks)
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/swift/llm/infer/infer_engine/infer_engine.py", line 115, in batch_run
return await asyncio.gather(*tasks)
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/swift/llm/infer/infer_engine/grpo_vllm_engine.py", line 170, in _new_run
res = await task
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/swift/plugin/multi_turn.py", line 86, in _infer_async_single
return await self.run(infer_request, request_config, **kwargs)
File "/checkpoint/binary/train_package/train/plugin.py", line 781, in run
return rollout_outputs if rollout_outputs else [RolloutOutput(
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/pydantic/main.py", line 250, in __init__
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for RolloutOutput
response
Input should be a dictionary or an instance of ChatCompletionResponse [type=dataclass_type, input_value=None, input_type=NoneType]
For further information visit https://errors.pydantic.dev/2.12/v/dataclass_type
0%| | 0/16 [00:30<?, ?it/s]
INFO: 33.107.100.54:36856 - "POST /infer/ HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
return await self.app(scope, receive, send)
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/fastapi/applications.py", line 1133, in __call__
await super().__call__(scope, receive, send)
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/starlette/applications.py", line 113, in __call__
await self.middleware_stack(scope, receive, send)
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
raise exc
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
await self.app(scope, receive, _send)
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
await self.app(scope, receive, send)
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/starlette/routing.py", line 716, in __call__
await self.middleware_stack(scope, receive, send)
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/starlette/routing.py", line 736, in app
await route.handle(scope, receive, send)
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/starlette/routing.py", line 290, in handle
await self.app(scope, receive, send)
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/fastapi/routing.py", line 123, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/fastapi/routing.py", line 109, in app
response = await f(request)
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/fastapi/routing.py", line 389, in app
raw_response = await run_endpoint_function(
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/fastapi/routing.py", line 288, in run_endpoint_function
return await dependant.call(**values)
File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/swift/llm/infer/rollout.py", line 590, in infer
all_outputs = list(chain.from_iterable(all_outputs)) # from list of list to single list
TypeError: 'NoneType' object is not iterable
rollout sh:
MODEL_PATH=Qwen/Qwen3-VL-32B-Instruct
export VLLM_RPC_TIMEOUT=500000
export VLLM_LOGGING_LEVEL=DEBUG
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
swift rollout \
--model $MODEL_PATH \
--infer_backend vllm \
--max_turns 3 \
--vllm_tensor_parallel_size 4 \
--vllm_data_parallel_size 2 \
--vllm_gpu_memory_utilization 0.5 \
--vllm_max_model_len 65536 \
--max_length 32768 \
--agent_template hermes \
--temperature 0.8 \
--padding_free true \
--attn_impl flash_attention_2 \
--multi_turn_scheduler our_scheduler \
--external_plugins train/plugin.py \
--vllm_limit_mm_per_prompt '{"image": 4, "video": 0}' \
--vllm_use_async_engine true \
--port 8000
grpo sh:
MODEL_PATH=Qwen/Qwen3-VL-32B-Instruct
export VLLM_LOGGING_LEVEL=DEBUG
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
NPROC_PER_NODE=$nproc_per_node \
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
NNODES=$nnodes \
NODE_RANK=$node_rank \
MASTER_ADDR=$master_addr \
MASTER_PORT=$master_port \
swift rlhf \
--rlhf_type grpo \
--model $MODEL_PATH \
--external_plugins train/plugin.py \
--multi_turn_scheduler our_scheduler \
--max_turns 3 \
--agent_template hermes \
--loss_scale hermes \
--reward_funcs our_reward \
--use_vllm true \
--vllm_mode server \
--vllm_server_host ${VLLM_HOST} \
--vllm_server_port ${VLLM_PORT} \
--vllm_server_timeout 300 \
--train_type full \
--torch_dtype bfloat16 \
--dataset 'train/data/grpo_train.jsonl' \
--load_from_cache_file true \
--max_completion_length 8192 \
--num_train_epochs 1 \
--per_device_train_batch_size 1 \
--learning_rate 1e-6 \
--gradient_accumulation_steps 1 \
--save_strategy 'steps' \
--save_steps 1000 \
--save_total_limit 10 \
--logging_steps 1 \
--output_dir output/models/Qwen3-VL-32B-Instruct/grpo \
--warmup_ratio 0.01 \
--dataloader_num_workers 1 \
--num_generations 8 \
--temperature 0.7 \
--padding_free true \
--attn_impl flash_attention_2 \
--deepspeed zero3 \
--move_model_batches 8 \
--add_version false \
--create_checkpoint_symlink true
Your hardware and system info
Write your system info like CUDA version/system/GPU/torch version here
Python version: 3.10.13+gc
NVCC version: Cuda compilation tools, release 12.8, V12.8.61
PyTorch CUDA version: 12.8
PyTorch cuDNN version: 91200
pip Package Version
--------------------------------- -----------------------
absl-py 2.3.1
accelerate 1.11.0
acm-sdk-python 0.4.11
addict 2.4.0
ai-scheduler 0.9.19
aiofiles 24.1.0
aiohappyeyeballs 2.6.1
aiohttp 3.13.1
aiooss2 0.2.11
aiosignal 1.4.0
aliyun-python-sdk-core 2.16.0
aliyun-python-sdk-kms 2.16.5
annotated-types 0.7.0
antlr4-python3-runtime 4.9.3
anyio 4.11.0
apache-tvm-ffi 0.1.0b15
apex 0.1
argon2-cffi 25.1.0
argon2-cffi-bindings 25.1.0
arrow 1.4.0
astor 0.8.1
asttokens 3.0.0
astunparse 1.6.3
async-lru 2.0.5
async-timeout 5.0.1
attrdict 2.0.1
attrs 25.4.0
av 16.0.1
babel 2.17.0
beautifulsoup4 4.14.2
binpacking 1.5.2
blake3 1.0.8
bleach 6.2.0
bracex 2.6
brotli 1.2.0
cachetools 6.2.1
cbor2 5.7.0
certifi 2025.10.5
cffi 2.0.0
chardet 3.0.4
charset-normalizer 3.4.4
click 8.2.1
cloudpickle 3.1.1
column-io 0.2.11+7c56d623.abi1
comm 0.2.3
common-io 0.8.23
compressed-tensors 0.11.0
contourpy 1.3.2
cpm-kernels 1.0.11
crcmod 1.7
cryptography 46.0.3
cuda-bindings 13.0.2
cuda-pathfinder 1.3.1
cuda-python 13.0.2
cupy-cuda12x 13.6.0
cycler 0.12.1
Cython 0.29.32
dacite 1.9.2
datasets 3.6.0
debugpy 1.8.17
decorator 5.2.1
decord 0.6.0
deepspeed 0.17.6
defusedxml 0.7.1
depyf 0.19.0
dill 0.3.8
diskcache 5.6.3
distro 1.9.0
dnspython 2.8.0
dockerfile-parse 2.0.1
docopt 0.6.2
e2b 2.8.1
e2b-code-interpreter 2.4.1
easydict 1.13
einops 0.8.1
email-validator 2.3.0
evaluate 0.4.6
exceptiongroup 1.3.0
executing 2.2.1
fastapi 0.119.1
fastapi-cli 0.0.14
fastapi-cloud-cli 0.3.1
fastjsonschema 2.21.2
fastrlock 0.8.3
ffmpy 1.0.0
filelock 3.20.0
filetype 1.2.0
flash_attn 2.8.0.post2
flash_attn_3 3.0.0b1
flash_mla 1.0.0+1408756
flashinfer-python 0.4.0
fonttools 4.60.1
fqdn 1.5.1
frozendict 2.4.7
frozenlist 1.8.0
fsspec 2025.3.0
future 0.18.2
gcsfs 2025.9.0
gguf 0.17.1
gitdb 4.0.12
GitPython 3.1.45
google-api-core 2.10.1
google-auth 2.41.1
google-auth-oauthlib 1.2.2
google-cloud-core 2.4.3
google-cloud-storage 2.14.0
google-crc32c 1.7.1
google-resumable-media 2.7.2
googleapis-common-protos 1.56.4
GPUtil 1.4.0
gradio 6.0.1
gradio_client 2.0.0
groovy 0.1.2
grouped_gemm 1.1.5
grpcio 1.75.1
h11 0.16.0
hdfs 2.7.3
hf-xet 1.1.10
hjson 3.1.0
httpcore 1.0.9
httptools 0.7.1
httpx 0.28.1
huggingface-hub 0.35.3
idna 2.8
ImageHash 4.3.1
importlib_metadata 8.7.0
intel-cmplr-lib-ur 2025.2.1
intel-openmp 2025.2.1
interegular 0.3.3
ipykernel 7.0.1
ipython 8.37.0
ipywidgets 8.1.7
isoduration 20.11.0
jedi 0.19.2
jieba 0.42.1
Jinja2 3.1.6
jiter 0.11.1
jmespath 0.10.0
joblib 1.5.2
json_repair 0.54.2
json5 0.12.1
jsonpointer 3.0.0
jsonschema 4.25.1
jsonschema-specifications 2025.9.1
jupyter 1.1.1
jupyter_client 8.6.3
jupyter-console 6.6.3
jupyter_core 5.9.1
jupyter-events 0.12.0
jupyter-lsp 2.3.0
jupyter_server 2.17.0
jupyter_server_terminals 0.5.3
jupyterlab 4.4.9
jupyterlab_pygments 0.3.0
jupyterlab_server 2.27.3
jupyterlab_widgets 3.0.15
kazoo 2.9.0
kiwisolver 1.4.9
kmontitor-client 0.0.0
lake_py_lib 0.1.16
lark 1.2.2
latex2sympy2_extended 1.0.6
lightning-utilities 0.15.2
llguidance 0.7.30
llvmlite 0.44.0
lm-format-enforcer 0.11.3
lmdb 0.94
loguru 0.7.3
lru-dict 1.3.0
Markdown 3.9
markdown-it-py 4.0.0
MarkupSafe 3.0.3
math-verify 0.5.2
matplotlib 3.9.2
matplotlib-inline 0.1.7
mdl 0.3.1
mdurl 0.1.2
minify_html 0.16.4
mistral_common 1.8.5
mistune 3.1.4
mizani 0.14.2
mkl 2025.0.1
mkl-include 2025.0.1
ml_dtypes 0.5.3
modelscope 1.32.0
mpmath 1.3.0
ms_swift 3.11.0.dev0
msgpack 1.1.2
msgspec 0.19.0
multidict 6.7.0
multimethod 1.12
multiprocess 0.70.16
nbclient 0.10.2
nbconvert 7.16.6
nbformat 5.10.4
nebula-mos-python-sdk 0.3.17
nebula-py-pangu-early-test 0.0.52
nest-asyncio 1.6.0
networkx 3.4.2
ninja 1.13.0
nltk 3.9.2
notebook 7.4.7
notebook_shim 0.2.4
numba 0.61.2
numpy 2.2.6
nvidia-cublas-cu12 12.8.4.1
nvidia-cuda-cupti-cu12 12.8.90
nvidia-cuda-nvrtc-cu12 12.8.93
nvidia-cuda-runtime-cu12 12.8.90
nvidia-cudnn-cu12 9.10.2.21
nvidia-cudnn-frontend 1.15.0
nvidia-cufft-cu12 11.3.3.83
nvidia-cufile-cu12 1.13.1.3
nvidia-curand-cu12 10.3.9.90
nvidia-cusolver-cu12 11.7.3.90
nvidia-cusparse-cu12 12.5.8.93
nvidia-cusparselt-cu12 0.7.1
nvidia-cutlass-dsl 4.2.1
nvidia-ml-py 12.575.51
nvidia-ml-py3 7.352.0
nvidia-nccl-cu12 2.27.3
nvidia-nvjitlink-cu12 12.8.93
nvidia-nvtx-cu12 12.8.90
oauthlib 3.3.1
omegaconf 2.3.0
onnx 1.19.1
onnx-ir 0.1.11
onnxscript 0.3.1
openai 2.6.0
openai-harmony 0.0.4
opencv-python 4.12.0.88
opencv-python-headless 4.12.0.88
openlm_hub 0.0.56
orjson 3.11.4
oss2 2.18.4
ossfs 2025.5.0
outlines_core 0.2.11
overrides 7.7.0
packaging 25.0
pandas 2.3.3
pandocfilters 1.5.1
pangudfs-client 1.0.27+lightsdk1.1.28
parso 0.8.5
partial-json-parser 0.2.1.1.post6
pathlib 1.0.1
patsy 1.0.2
peft 0.18.0
pexpect 4.9.0
phik 0.12.5
pillow 10.4.0
pip 24.0
platformdirs 4.5.0
plotnine 0.15.0
prometheus_client 0.23.1
prometheus-fastapi-instrumentator 7.1.0
prompt_toolkit 3.0.52
propcache 0.4.1
protobuf 6.33.1
psutil 5.9.5
ptyprocess 0.7.0
pure_eval 0.2.3
puremagic 1.30
py-cpuinfo 9.0.0
py-spy 0.4.1
pyarrow 21.0.0
pyasn1 0.6.1
pyasn1_modules 0.4.2
pybase64 1.4.2
pybind11 3.0.1
pycountry 24.6.1
pycparser 2.23
pycryptodome 3.23.0
pydantic 2.12.3
pydantic_core 2.41.4
pydantic-extra-types 2.10.6
pydicom 1.2.2
pydub 0.25.1
Pygments 2.19.2
pykmonitor 2.2.0
pynvml 12.0.0
pyodps 0.12.3
pyparsing 3.2.5
pyreft 0.0.8
pytest-runner 6.0.1
python-dateutil 2.9.0.post0
python-dotenv 1.1.1
python-json-logger 4.0.0
python-multipart 0.0.20
pytz 2025.2
pyvene 0.1.8
PyWavelets 1.8.0
PyYAML 6.0.3
pyzmq 27.1.0
qwen-vl-utils 0.0.14
ray 2.50.1
redis 6.4.0
referencing 0.37.0
regex 2025.10.22
requests 2.32.5
requests-oauthlib 2.0.0
retrying 1.4.2
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rfc3987-syntax 1.1.0
rich 14.2.0
rich-toolkit 0.15.1
rignore 0.7.1
rouge 1.0.1
rpds-py 0.27.1
rsa 4.9.1
ruamel.yaml 0.18.15
ruamel.yaml.clib 0.2.14
safehttpx 0.1.7
safetensors 0.6.2
scikit-learn 1.7.2
scipy 1.14.0
seaborn 0.12.2
semantic-version 2.10.0
Send2Trash 1.8.3
sentencepiece 0.2.1
sentry-sdk 2.42.1
setproctitle 1.3.7
setuptools 80.9.0
shellingham 1.5.4
simplejson 3.17.6
six 1.14.0
sklearn 0.0.post12
smart_open 7.4.0
smmap 5.0.2
sniffio 1.3.1
sortedcontainers 2.4.0
soundfile 0.13.1
soupsieve 2.8
soxr 1.0.0
stack-data 0.6.3
starlette 0.48.0
statsmodels 0.14.5
sympy 1.14.0
tabulate 0.9.0
tbb 2022.2.0
tcmlib 1.4.0
tensorboard 2.20.0
tensorboard-data-server 0.7.2
terminado 0.18.1
thop-statistics 0.1.1.post2303141613
threadpoolctl 3.6.0
thrift 0.16.0
tiktoken 0.12.0
tinycss2 1.4.0
tokenizers 0.22.1
tomli 2.3.0
tomlkit 0.13.3
torch 2.8.0
torchaudio 2.8.0
torchdata 0.11.0
torchmetrics 1.8.2
torchvision 0.23.0
tornado 6.1
tqdm 4.67.1
traitlets 5.14.3
transformer_engine_mdl 2.6.0+torch2.8.bf78e5d1
transformers 4.57.1
transformers-stream-generator 0.0.5
transitions 0.9.0
triton 3.4.0
trl 0.24.0
typeguard 4.4.4
typer 0.20.0
typing_extensions 4.15.0
typing-inspection 0.4.2
tzdata 2025.2
umf 0.11.0
uri-template 1.3.0
urllib3 2.5.0
uvicorn 0.38.0
uvloop 0.22.1
visions 0.8.1
vllm 0.11.0
wandb 0.22.2
watchfiles 1.1.1
wcmatch 10.1
wcwidth 0.2.14
webcolors 24.11.1
webencodings 0.5.1
websocket-client 1.9.0
websockets 15.0.1
Werkzeug 3.1.3
wheel 0.45.1
widgetsnbextension 4.0.14
wordcloud 1.9.4
wrapt 2.0.0
xformers 0.0.32.post1
xgrammar 0.1.25
xxhash 3.6.0
yarl 1.22.0
ydata-profiling 4.17.0
zipp 3.23.0
zstandard 0.25.0
Additional context
Add any other context about the problem here
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the server-mode GRPO setup with --vllm_use_async_engine true, then rerun with CUDA_LAUNCH_BLOCKING=1 as the traceback suggests. Trace from vllm/v1/worker/gpu_model_runner.py through vllm/model_executor/layers/logits_processor.py and the distributed all_gather path; done means isolating whether the failure is in this path or the custom multimodal scheduler and documenting a minimal reproducer.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 18/100