huggingface / huggingface/trl

Multimodal generation hangs in vllm server mode with large images (works in colocate), manual resizing required

Open
#5,361 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.3k
Forks
3k
Avg merge
1d 20h
Merged PRs (30d)
194

Description

### Reproduction

Description:
When running a multimodal RL training script (e.g., GRPO with Qwen-VL) using TRL and vLLM, I encountered a severe hanging issue specific to the vllm-mode="server".

If I pass original high-resolution images (e.g., 1080x720, 1555x709) to the environment, the generation process completely hangs/freezes in server mode. However, the exact same images process perfectly fine without any issues when using vllm-mode="colocate".

The Workaround:
The only way to prevent the server mode from hanging is to manually resize the images before passing them to the messages:

```
Python
from PIL import Image

MAX_DIM = 1024
# ...
if max(img.size) > MAX_DIM:
img.thumbnail((MAX_DIM, MAX_DIM))
```
Logs showing the problematic image sizes (before resizing):

```
Plaintext
Sample 'WIT_184' Step 0 | Multiple Prompt Length: 4036 tokens | Text Prompt Length: 2212 | Images: 2
- Image size: (1080, 720), mode: RGB
- Image size: (1555, 709), mode: RGB
```
Expected Behavior:
I suspect the hang in server mode is caused by the massive payload size of Base64 encoded images blocking the JSON serialization or causing an HTTP timeout, which colocate mode avoids by sharing memory/pointers.

However, as a user, I do not want to manually downsample my images, as this loss of resolution might severely degrade the model's visual performance. The framework should ideally handle standard high-res images natively in server mode without freezing (perhaps by optimizing the image transmission protocol, supporting local file path references, or handling the preprocessing dynamically).

### System Info

- Platform: Linux-5.4.0-204-generic-x86_64-with-glibc2.31
- Python version: 3.12.12
- TRL version: 1.0.0.dev0
- PyTorch version: 2.9.1
- accelerator(s): NVIDIA A100-SXM4-80GB, NVIDIA A100-SXM4-80GB, NVIDIA A100-SXM4-80GB, NVIDIA A100-SXM4-80GB, NVIDIA A100-SXM4-80GB, NVIDIA A100-SXM4-80GB, NVIDIA A100-SXM4-80GB, NVIDIA A100-SXM4-80GB
- Transformers version: 4.57.6
- Accelerate version: 1.13.0
- Accelerate config:
- compute_environment: LOCAL_MACHINE
- distributed_type: DEEPSPEED
- mixed_precision: bf16
- use_cpu: False
- debug: True
- num_processes: 2
- machine_rank: 0
- num_machines: 1
- rdzv_backend: static
- same_network: True
- main_training_function: main
- enable_cpu_affinity: False
- deepspeed_config: {'gradient_accumulation_steps': 4, 'offload_optimizer_device': 'none', 'offload_param_device': 'none', 'zero3_init_flag': True, 'zero_stage': 2}
- downcast_bf16: no
- tpu_use_cluster: False
- tpu_use_sudo: False
- tpu_env: []
- dynamo_config: {'dynamo_backend': 'EAGER', 'dynamo_mode': 'default', 'dynamo_use_dynamic': False, 'dynamo_use_fullgraph': False, 'dynamo_use_regional_compilation': False}
- Datasets version: 4.6.1
- HF Hub version: 0.36.2
- bitsandbytes version: 0.49.2
- DeepSpeed version: 0.18.8
- Liger-Kernel version: not installed
- LLM-Blender version: not installed
- OpenAI version: 2.26.0
- PEFT version: 0.18.1
- vLLM version: 0.14.1

### Checklist

- [x] I have checked that my issue isn't already filed (see [open issues](https://github.com/huggingface/trl/issues?q=is%3Aissue))
- [x] I have included my system information
- [x] Any code provided is minimal, complete, and reproducible ([more on MREs](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks))
- [x] Any code provided is properly formatted in code blocks, (no screenshot, [more on code blocks](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks))
- [x] Any traceback provided is complete

Contributor guide

Open the contributing guide

Research direction

No repository file or test is named. Start by reproducing generation with the reported high-resolution images in vllm-mode="server" and compare it with colocate mode, focusing on the image request path and the reported hang. Done means server mode completes without requiring manual image resizing.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.