Multimodal generation hangs in vllm server mode with large images (works in colocate), manual resizing required
- Dominant language
- Python
- Stars
- 19.3k
- Forks
- 3k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 194
Description
### Reproduction
Description:
When running a multimodal RL training script (e.g., GRPO with Qwen-VL) using TRL and vLLM, I encountered a severe hanging issue specific to the vllm-mode="server".
If I pass original high-resolution images (e.g., 1080x720, 1555x709) to the environment, the generation process completely hangs/freezes in server mode. However, the exact same images process perfectly fine without any issues when using vllm-mode="colocate".
The Workaround:
The only way to prevent the server mode from hanging is to manually resize the images before passing them to the messages:
```
Python
from PIL import Image
MAX_DIM = 1024
# ...
if max(img.size) > MAX_DIM:
img.thumbnail((MAX_DIM, MAX_DIM))
```
Logs showing the problematic image sizes (before resizing):
```
Plaintext
Sample 'WIT_184' Step 0 | Multiple Prompt Length: 4036 tokens | Text Prompt Length: 2212 | Images: 2
- Image size: (1080, 720), mode: RGB
- Image size: (1555, 709), mode: RGB
```
Expected Behavior:
I suspect the hang in server mode is caused by the massive payload size of Base64 encoded images blocking the JSON serialization or causing an HTTP timeout, which colocate mode avoids by sharing memory/pointers.
However, as a user, I do not want to manually downsample my images, as this loss of resolution might severely degrade the model's visual performance. The framework should ideally handle standard high-res images natively in server mode without freezing (perhaps by optimizing the image transmission protocol, supporting local file path references, or handling the preprocessing dynamically).
### System Info
- Platform: Linux-5.4.0-204-generic-x86_64-with-glibc2.31
- Python version: 3.12.12
- TRL version: 1.0.0.dev0
- PyTorch version: 2.9.1
- accelerator(s): NVIDIA A100-SXM4-80GB, NVIDIA A100-SXM4-80GB, NVIDIA A100-SXM4-80GB, NVIDIA A100-SXM4-80GB, NVIDIA A100-SXM4-80GB, NVIDIA A100-SXM4-80GB, NVIDIA A100-SXM4-80GB, NVIDIA A100-SXM4-80GB
- Transformers version: 4.57.6
- Accelerate version: 1.13.0
- Accelerate config:
- compute_environment: LOCAL_MACHINE
- distributed_type: DEEPSPEED
- mixed_precision: bf16
- use_cpu: False
- debug: True
- num_processes: 2
- machine_rank: 0
- num_machines: 1
- rdzv_backend: static
- same_network: True
- main_training_function: main
- enable_cpu_affinity: False
- deepspeed_config: {'gradient_accumulation_steps': 4, 'offload_optimizer_device': 'none', 'offload_param_device': 'none', 'zero3_init_flag': True, 'zero_stage': 2}
- downcast_bf16: no
- tpu_use_cluster: False
- tpu_use_sudo: False
- tpu_env: []
- dynamo_config: {'dynamo_backend': 'EAGER', 'dynamo_mode': 'default', 'dynamo_use_dynamic': False, 'dynamo_use_fullgraph': False, 'dynamo_use_regional_compilation': False}
- Datasets version: 4.6.1
- HF Hub version: 0.36.2
- bitsandbytes version: 0.49.2
- DeepSpeed version: 0.18.8
- Liger-Kernel version: not installed
- LLM-Blender version: not installed
- OpenAI version: 2.26.0
- PEFT version: 0.18.1
- vLLM version: 0.14.1
### Checklist
- [x] I have checked that my issue isn't already filed (see [open issues](https://github.com/huggingface/trl/issues?q=is%3Aissue))
- [x] I have included my system information
- [x] Any code provided is minimal, complete, and reproducible ([more on MREs](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks))
- [x] Any code provided is properly formatted in code blocks, (no screenshot, [more on code blocks](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks))
- [x] Any traceback provided is complete
Contributor guide
Research direction
No repository file or test is named. Start by reproducing generation with the reported high-resolution images in vllm-mode="server" and compare it with colocate mode, focusing on the image request path and the reported hang. Done means server mode completes without requiring manual image resizing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100