alibaba / alibaba/ROLL

[Bug] Training/Inference crash with Qwen2/3-VL due to missing mm_token_type_ids in Collator

Open
#435 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.4k
Forks
312
Avg merge
1h 2m
Merged PRs (30d)
2

Description

## Describe the bug
Currently, when using the ROLL framework to train or sample with latest multi-modal models like Qwen2-VL or Qwen3-VL, the system crashes during the data collection or forward pass.

The issue stems from two missing components in the core library:
1. Collator Support: The DataCollatorWithPaddingForMM does not recognize or pad the mm_token_type_ids field produced by the Qwen Processor. This causes tensor creation errors or dimension mismatches when batching interleaved multimodal data.
2. Positional Encoding Binding: Specifically for Qwen3-VL, the necessary dynamic methods for vision position ID generation are not bound in model_providers.py, leading to a KeyError when calculating 3D RoPE indices.

## To Reproduce
Steps to reproduce the behavior:
1. Initialize an RLVR or SFT pipeline using Qwen3-VL-8B-Base.
2. Use a multimodal dataset with images.
3. The pipeline will crash at the first sampling or training step.

## Logs/Screenshots (Captured from real environment)

1 ray.exceptions.RayTaskError(AttributeError): ray::DynamicSamplingScheduler.get_batch() (pid=310859, ip=10.83.115.19)
2 File ".../roll/distributed/scheduler/generate_scheduler.py", line 351, in get_batch
3 batch = self.collator(samples)
4 File ".../roll/datasets/collator.py", line 180, in __call__
5 # KeyError triggered during model-specific rope index generation
6 KeyError: 'get_vision_position_ids'

## Environment:
- Hardware: NVIDIA H200
- Model: Qwen3-VL / Qwen2-VL
- Transformers Version: 4.45.0+ (and latest)

## Additional context
Without proper handling of mm_token_type_ids, the 3D RoPE (Rotary Positional Embedding) cannot be calculated correctly for multi-image batches, resulting in either a crash or model output degradation.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with roll/datasets/collator.py, especially DataCollatorWithPaddingForMM and the failing call shown in the traceback, then inspect model_providers.py for Qwen3-VL vision position ID binding. Reproduce with a Qwen2/3-VL multimodal RLVR or SFT pipeline and image data. Done means batching handles mm_token_type_ids and Qwen3-VL no longer raises the get_vision_position_ids KeyError during sampling or training.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.