Support image paths and URLs in vLLM generation
- Dominant language
- Python
- Stars
- 19.3k
- Forks
- 3k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 194
Description
### Problem
VLM prompts can contain image sources as local paths or URLs, and the model processor accepts those sources when applying the chat template. `GRPOTrainer` and `RLOOTrainer` currently extract the raw `image` values and pass them to `VLLMGeneration` unchanged.
That breaks the vLLM path:
- server mode serializes each item with `PIL.Image.save`, so a string path fails;
- colocated mode forwards the string as `multi_modal_data`, instead of the processor-loaded image used by the normal generation path.
A minimal prompt is:
```python
prompt = [{
"role": "user",
"content": [
{"type": "image", "image": "/path/to/image.png"},
{"type": "text", "text": "Describe this image."},
],
}]
```
with `GRPOConfig(use_vllm=True)`.
### Expected behavior
The shared vLLM generation boundary should accept the same image-source forms as Transformers processors (PIL images, local paths, and URLs), normalize them through the configured image processor, and then pass PIL images to either vLLM mode.
I am preparing a small fix and regression test at the shared `VLLMGeneration` boundary so GRPO, RLOO, server mode, and colocated mode receive the same behavior.
Contributor guide
Research direction
Start at the shared VLLMGeneration boundary and inspect how the configured image processor handles PIL images, local paths, and URLs. Add the mentioned regression coverage for GRPO and RLOO behavior in both server and colocated modes; done means all supported image-source forms reach vLLM as PIL images.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100