Qwen3-VL-Embedding, some problems when training videos

When training Qwen3-VL-Embedding with video inputs, I encountered the following two issues related to video input handling.

1. For data in the following format:
   { "messages": [ { "role": "user", "content": "<video> Find videos that contain flags." } ], "videos": [ ["/phd/xuhanwen/data/Timelens-clip/cosmo_cap/Y9nw5LcC4N4/Y9nw5LcC4N4_seg13.mp4"] ] }
   The following error occurs:

Traceback (most recent call last):
File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/site-packages/swift/llm/dataset/utils.py", line 102, in getitem
return self.encode_func(data, return_length=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/site-packages/swift/utils/utils.py", line 486, in new_func
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/site-packages/swift/llm/template/base.py", line 539, in encode
encoded = self._embedding_encode(inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/site-packages/swift/llm/template/base.py", line 403, in _embedding_encode
anchor_encoded = self._encode_truncated(anchor)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/site-packages/swift/llm/template/base.py", line 1281, in _encode_truncated
encoded = self._encode(inputs)
^^^^^^^^^^^^^^^^^^^^
File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/site-packages/swift/llm/template/template/qwen.py", line 484, in _encode
encoded = Template._encode(self, inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/site-packages/swift/llm/template/base.py", line 1348, in _encode
res_context_list, loss_scale_list = self._simplify_context_list(res_context_list, loss_scale_list, inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/site-packages/swift/llm/template/base.py", line 743, in _simplify_context_list
context_list, loss_scale_list = self._pre_tokenize(context_list, loss_scale_list, inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/site-packages/swift/llm/template/base.py", line 915, in _pre_tokenize
c_list = self.replace_tag(k, getattr(inputs, f'{k}_idx'), inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/site-packages/swift/llm/template/template/qwen.py", line 308, in replace_tag
video, video_kwargs = fetch_video(video_inputs, return_video_sample_fps=True, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/site-packages/qwen_vl_utils/vision_process.py", line 434, in fetch_video
image_list = [future.result() for future in futures]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/site-packages/qwen_vl_utils/vision_process.py", line 434, in
image_list = [future.result() for future in futures]
^^^^^^^^^^^^^^^
File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/site-packages/qwen_vl_utils/vision_process.py", line 117, in fetch_image
image_obj = Image.open(image)
^^^^^^^^^^^^^^^^^
File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/site-packages/PIL/Image.py", line 3579, in open
raise UnidentifiedImageError(msg)
PIL.UnidentifiedImageError: cannot identify image file '/phd/xuhanwen/data/Timelens-clip/cosmo_cap/Y9nw5LcC4N4/Y9nw5LcC4N4_seg13.mp4'

2. For data in the following format:
   { "messages": [ { "role": "user", "content": "<video> Find videos that contain flags." } ], "videos": ["/phd/xuhanwen/data/Timelens-clip/cosmo_cap/Y9nw5LcC4N4/Y9nw5LcC4N4_seg13.mp4"] }
   The following error occurs:

[rank5]: Original Traceback (most recent call last):
[rank5]: File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/site-packages/torch/utils/data/_utils/worker.py", line 349, in _worker_loop
[rank5]: data = fetcher.fetch(index) # type: ignore[possibly-undefined]
[rank5]: ^^^^^^^^^^^^^^^^^^^^
[rank5]: File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/site-packages/torch/utils/data/_utils/fetch.py", line 55, in fetch
[rank5]: return self.collate_fn(data)
[rank5]: ^^^^^^^^^^^^^^^^^^^^^
[rank5]: File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/site-packages/swift/llm/template/base.py", line 1515, in data_collator
[rank5]: res = self._embedding_data_collator(batch, padding_to=padding_to)
[rank5]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]: File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/site-packages/swift/llm/template/base.py", line 1628, in _embedding_data_collator
[rank5]: res = self._data_collator(new_batch, padding_to=padding_to)
[rank5]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]: File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/site-packages/swift/llm/template/template/qwen.py", line 444, in _data_collator
[rank5]: res['position_ids'] = self._get_position_ids(res)
[rank5]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]: File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/site-packages/swift/llm/template/template/qwen.py", line 433, in _get_position_ids
[rank5]: position_ids, _ = get_rope_index(
[rank5]: ^^^^^^^^^^^^^^^
[rank5]: File "/data/xuhanwen05/envs/swift/swift/lib/python3.11/site-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py", line 965, in get_rope_index
[rank5]: ed_video = input_tokens.index(video_token_id, st)
[rank5]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]: ValueError: 151656 is not in list

How exactly is Qwen3-VL-Embedding supposed to be trained with video inputs for contrastive learning?

Start by reproducing both JSON formats with the Qwen3-VL-Embedding training path. Inspect swift/llm/template/template/qwen.py, swift/llm/template/base.py, qwen_vl_utils/vision_process.py, and the Qwen3-VL get_rope_index traceback. Done means video inputs work for contrastive training without the image-decoding or missing-video-token errors.