huggingface / huggingface/trl

Support image paths and URLs in vLLM generation

Open
#6,519 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.3k
Forks
3k
Avg merge
1d 20h
Merged PRs (30d)
194

Description

### Problem

VLM prompts can contain image sources as local paths or URLs, and the model processor accepts those sources when applying the chat template. `GRPOTrainer` and `RLOOTrainer` currently extract the raw `image` values and pass them to `VLLMGeneration` unchanged.

That breaks the vLLM path:

- server mode serializes each item with `PIL.Image.save`, so a string path fails;
- colocated mode forwards the string as `multi_modal_data`, instead of the processor-loaded image used by the normal generation path.

A minimal prompt is:

```python
prompt = [{
"role": "user",
"content": [
{"type": "image", "image": "/path/to/image.png"},
{"type": "text", "text": "Describe this image."},
],
}]
```

with `GRPOConfig(use_vllm=True)`.

### Expected behavior

The shared vLLM generation boundary should accept the same image-source forms as Transformers processors (PIL images, local paths, and URLs), normalize them through the configured image processor, and then pass PIL images to either vLLM mode.

I am preparing a small fix and regression test at the shared `VLLMGeneration` boundary so GRPO, RLOO, server mode, and colocated mode receive the same behavior.

Contributor guide

Open the contributing guide

Research direction

Start at the shared VLLMGeneration boundary and inspect how the configured image processor handles PIL images, local paths, and URLs. Add the mentioned regression coverage for GRPO and RLOO behavior in both server and colocated modes; done means all supported image-source forms reach vLLM as PIL images.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.