modelscope / modelscope/ms-swift

使用PtEngine推理Qwen3VL，当使用adapter_request时报错

Open
#6,873 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stale
Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

Describe the bug
使用PtEngine推理Qwen3VL，当使用adapter_request时报错。看样子是不适配vision encoder

---------------------------------------------------------------------------
File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/swift/llm/infer/infer_engine/pt_engine.py:591, in PtEngine.infer(self, infer_requests, request_config, metrics, template, use_tqdm, adapter_request)
    589 while i < len(infer_requests):
    590     infer_requests_samples = infer_requests[i:i + max_batch_size]
--> [591](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/swift/llm/infer/infer_engine/pt_engine.py:591)     res += self._infer(
    592         infer_requests_samples, request_config, template=template, adapter_request=adapter_request)
    593     i += max_batch_size
    594     prog_bar.update(len(infer_requests_samples))

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/utils/_contextlib.py:120, in context_decorator.<locals>.decorate_context(*args, **kwargs)
    117 @functools.wraps(func)
    118 def decorate_context(*args, **kwargs):
    119     with ctx_factory():
--> [120](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/utils/_contextlib.py:120)         return func(*args, **kwargs)

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/swift/llm/infer/infer_engine/pt_engine.py:554, in PtEngine._infer(self, infer_requests, request_config, template, adapter_request, pre_infer_hook)
    550 if len(kwargs) > 0:
    551     infer_func = self._infer_forward if template.task_type in {
    552         'seq_cls', 'prm', 'embedding', 'reranker', 'generative_reranker'
    553     } else self._infer_full
--> [554](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/swift/llm/infer/infer_engine/pt_engine.py:554)     res = infer_func(**kwargs)
    555 else:
    556     res = []

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/swift/llm/infer/infer_engine/pt_engine.py:398, in PtEngine._infer_full(self, template, inputs, generation_config, adapter_request, request_config, template_inputs)
    396 num_prompt_tokens = self._get_num_tokens(inputs)
    397 generate_kwargs = template.prepare_generate_kwargs(generate_kwargs, model=self.model)
--> [398](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/swift/llm/infer/infer_engine/pt_engine.py:398) output = dict(template.generate(self.model, **generate_kwargs))
    399 output.pop('past_key_values', None)
    400 batched_generate_ids = output['sequences']

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/swift/llm/template/base.py:639, in Template.generate(self, model, *args, **kwargs)
    637 if 'use_model_defaults' in signature.parameters and 'use_model_defaults' not in kwargs:
    638     kwargs['use_model_defaults'] = False
--> [639](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/swift/llm/template/base.py:639) return model.generate(*args, **kwargs)

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/peft/peft_model.py:1973, in PeftModelForCausalLM.generate(self, *args, **kwargs)
   1971     with self._enable_peft_forward_hooks(*args, **kwargs):
   1972         kwargs = {k: v for k, v in kwargs.items() if k not in self.special_peft_forward_args}
-> [1973](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/peft/peft_model.py:1973)         outputs = self.base_model.generate(*args, **kwargs)
   1974 else:
   1975     outputs = self.base_model.generate(**kwargs)

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/utils/_contextlib.py:120, in context_decorator.<locals>.decorate_context(*args, **kwargs)
    117 @functools.wraps(func)
    118 def decorate_context(*args, **kwargs):
    119     with ctx_factory():
--> [120](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/utils/_contextlib.py:120)         return func(*args, **kwargs)

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/transformers/generation/utils.py:2564, in GenerationMixin.generate(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, assistant_model, streamer, negative_prompt_ids, negative_prompt_attention_mask, use_model_defaults, custom_generate, **kwargs)
   2561 model_kwargs["use_cache"] = generation_config.use_cache
   2563 # 9. Call generation mode
-> [2564](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/transformers/generation/utils.py:2564) result = decoding_method(
   2565     self,
   2566     input_ids,
   2567     logits_processor=prepared_logits_processor,
   2568     stopping_criteria=prepared_stopping_criteria,
   2569     generation_config=generation_config,
   2570     **generation_mode_kwargs,
   2571     **model_kwargs,
   2572 )
   2574 # Convert to legacy cache format if requested
   2575 if (
   2576     generation_config.return_legacy_cache is True
   2577     and hasattr(result, "past_key_values")
   2578     and getattr(result.past_key_values, "to_legacy_cache") is not None
   2579 ):

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/transformers/generation/utils.py:2784, in GenerationMixin._sample(self, input_ids, logits_processor, stopping_criteria, generation_config, synced_gpus, streamer, **model_kwargs)
   2781 model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
   2783 if is_prefill:
-> [2784](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/transformers/generation/utils.py:2784)     outputs = self(**model_inputs, return_dict=True)
   2785     is_prefill = False
   2786 else:

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1773, in Module._wrapped_call_impl(self, *args, **kwargs)
   1771     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1772 else:
-> [1773](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1773)     return self._call_impl(*args, **kwargs)

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1784, in Module._call_impl(self, *args, **kwargs)
   1779 # If we don't have any hooks, we want to skip the rest of the logic in
   1780 # this function, and just call forward.
   1781 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1782         or _global_backward_pre_hooks or _global_backward_hooks
   1783         or _global_forward_hooks or _global_forward_pre_hooks):
-> [1784](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1784)     return forward_call(*args, **kwargs)
   1786 result = None
   1787 called_always_called_hooks = set()

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/transformers/utils/generic.py:1064, in check_model_inputs.<locals>.wrapper(self, *args, **kwargs)
   1061                 monkey_patched_layers.append((module, original_forward))
   1063 try:
-> [1064](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/transformers/utils/generic.py:1064)     outputs = func(self, *args, **kwargs)
   1065 except TypeError as original_exception:
   1066     # If we get a TypeError, it's possible that the model is not receiving the recordable kwargs correctly.
   1067     # Get a TypeError even after removing the recordable kwargs -> re-raise the original exception
   1068     # Otherwise -> we're probably missing `**kwargs` in the decorated function
   1069     kwargs_without_recordable = {k: v for k, v in kwargs.items() if k not in recordable_keys}

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py:1345, in Qwen3VLForConditionalGeneration.forward(self, input_ids, attention_mask, position_ids, past_key_values, inputs_embeds, labels, pixel_values, pixel_values_videos, image_grid_thw, video_grid_thw, cache_position, logits_to_keep, **kwargs)
   1314 @check_model_inputs
   1315 def forward(
   1316     self,
   (...)
   1329     **kwargs: Unpack[TransformersKwargs],
   1330 ) -> Union[tuple, Qwen3VLCausalLMOutputWithPast]:
   1331     r"""
   1332     labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
   1333         Labels for computing the masked language modeling loss. Indices should either be in `[0, ...,
   (...)
   1342         TODO: Add example
   1343     """
-> [1345](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py:1345)     outputs = self.model(
   1346         input_ids=input_ids,
   1347         pixel_values=pixel_values,
   1348         pixel_values_videos=pixel_values_videos,
   1349         image_grid_thw=image_grid_thw,
   1350         video_grid_thw=video_grid_thw,
   1351         position_ids=position_ids,
   1352         attention_mask=attention_mask,
   1353         past_key_values=past_key_values,
   1354         inputs_embeds=inputs_embeds,
   1355         cache_position=cache_position,
   1356         **kwargs,
   1357     )
   1359     hidden_states = outputs[0]
   1361     # Only compute necessary logits, and do not upcast them to float if we are not computing the loss

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:[1773](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1773), in Module._wrapped_call_impl(self, *args, **kwargs)
   1771     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1772 else:
-> 1773     return self._call_impl(*args, **kwargs)

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1784, in Module._call_impl(self, *args, **kwargs)
   1779 # If we don't have any hooks, we want to skip the rest of the logic in
   1780 # this function, and just call forward.
   1781 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1782         or _global_backward_pre_hooks or _global_backward_hooks
   1783         or _global_forward_hooks or _global_forward_pre_hooks):
-> [1784](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1784)     return forward_call(*args, **kwargs)
   1786 result = None
   1787 called_always_called_hooks = set()

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/transformers/utils/generic.py:1064, in check_model_inputs.<locals>.wrapper(self, *args, **kwargs)
   1061                 monkey_patched_layers.append((module, original_forward))
   1063 try:
-> [1064](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/transformers/utils/generic.py:1064)     outputs = func(self, *args, **kwargs)
   1065 except TypeError as original_exception:
   1066     # If we get a TypeError, it's possible that the model is not receiving the recordable kwargs correctly.
   1067     # Get a TypeError even after removing the recordable kwargs -> re-raise the original exception
   1068     # Otherwise -> we're probably missing `**kwargs` in the decorated function
   1069     kwargs_without_recordable = {k: v for k, v in kwargs.items() if k not in recordable_keys}

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py:1146, in Qwen3VLModel.forward(self, input_ids, attention_mask, position_ids, past_key_values, inputs_embeds, pixel_values, pixel_values_videos, image_grid_thw, video_grid_thw, cache_position, **kwargs)
   1143     inputs_embeds = inputs_embeds.masked_scatter(image_mask, image_embeds)
   1145 if pixel_values_videos is not None:
-> [1146](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py:1146)     video_embeds, deepstack_video_embeds = self.get_video_features(pixel_values_videos, video_grid_thw)
   1147     video_embeds = torch.cat(video_embeds, dim=0).to(inputs_embeds.device, inputs_embeds.dtype)
   1148     _, video_mask = self.get_placeholder_mask(
   1149         input_ids, inputs_embeds=inputs_embeds, video_features=video_embeds
   1150     )

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py:1048, in Qwen3VLModel.get_video_features(self, pixel_values_videos, video_grid_thw)
   1038 """
   1039 Encodes videos into continuous embeddings that can be forwarded to the language model. The deepstack visual features are also returned.
   1040 
   (...)
   1045         The temporal, height and width of feature shape of each video in LLM.
   1046 """
   1047 # Same implementation as for images
-> [1048](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py:1048) return self.get_image_features(pixel_values_videos, video_grid_thw)

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py:1061, in Qwen3VLModel.get_image_features(self, pixel_values, image_grid_thw)
   1051 """
   1052 Encodes images into continuous embeddings that can be forwarded to the language model. The deepstack visual features are also returned.
   1053 
   (...)
   1058         The temporal, height and width of feature shape of each image in LLM.
   1059 """
   1060 pixel_values = pixel_values.type(self.visual.dtype)
-> [1061](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py:1061) image_embeds, deepstack_image_embeds = self.visual(pixel_values, grid_thw=image_grid_thw)
   1062 split_sizes = (image_grid_thw.prod(-1) // self.visual.spatial_merge_size**2).tolist()
   1063 image_embeds = torch.split(image_embeds, split_sizes)

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1773, in Module._wrapped_call_impl(self, *args, **kwargs)
   1771     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1772 else:
-> [1773](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1773)     return self._call_impl(*args, **kwargs)

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1784, in Module._call_impl(self, *args, **kwargs)
   1779 # If we don't have any hooks, we want to skip the rest of the logic in
   1780 # this function, and just call forward.
   1781 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1782         or _global_backward_pre_hooks or _global_backward_hooks
   1783         or _global_forward_hooks or _global_forward_pre_hooks):
-> [1784](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1784)     return forward_call(*args, **kwargs)
   1786 result = None
   1787 called_always_called_hooks = set()

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py:739, in Qwen3VLVisionModel.forward(self, hidden_states, grid_thw, **kwargs)
    737 deepstack_feature_lists = []
    738 for layer_num, blk in enumerate(self.blocks):
--> [739](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py:739)     hidden_states = blk(
    740         hidden_states,
    741         cu_seqlens=cu_seqlens,
    742         position_embeddings=position_embeddings,
    743         **kwargs,
    744     )
    745     if layer_num in self.deepstack_visual_indexes:
    746         deepstack_feature = self.deepstack_merger_list[self.deepstack_visual_indexes.index(layer_num)](
    747             hidden_states
    748         )

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/transformers/modeling_layers.py:94, in GradientCheckpointingLayer.__call__(self, *args, **kwargs)
     91         logger.warning_once(message)
     93     return self._gradient_checkpointing_func(partial(super().__call__, **kwargs), *args)
---> [94](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/transformers/modeling_layers.py:94) return super().__call__(*args, **kwargs)

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1773, in Module._wrapped_call_impl(self, *args, **kwargs)
   1771     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1772 else:
-> [1773](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1773)     return self._call_impl(*args, **kwargs)

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1784, in Module._call_impl(self, *args, **kwargs)
   1779 # If we don't have any hooks, we want to skip the rest of the logic in
   1780 # this function, and just call forward.
   1781 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1782         or _global_backward_pre_hooks or _global_backward_hooks
   1783         or _global_forward_hooks or _global_forward_pre_hooks):
-> [1784](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1784)     return forward_call(*args, **kwargs)
   1786 result = None
   1787 called_always_called_hooks = set()

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py:267, in Qwen3VLVisionBlock.forward(self, hidden_states, cu_seqlens, rotary_pos_emb, position_embeddings, **kwargs)
    259 def forward(
    260     self,
    261     hidden_states: torch.Tensor,
   (...)
    265     **kwargs,
    266 ) -> torch.Tensor:
--> [267](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py:267)     hidden_states = hidden_states + self.attn(
    268         self.norm1(hidden_states),
    269         cu_seqlens=cu_seqlens,
    270         rotary_pos_emb=rotary_pos_emb,
    271         position_embeddings=position_embeddings,
    272         **kwargs,
    273     )
    274     hidden_states = hidden_states + self.mlp(self.norm2(hidden_states))
    275     return hidden_states

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1773, in Module._wrapped_call_impl(self, *args, **kwargs)
   1771     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1772 else:
-> [1773](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1773)     return self._call_impl(*args, **kwargs)

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1784, in Module._call_impl(self, *args, **kwargs)
   1779 # If we don't have any hooks, we want to skip the rest of the logic in
   1780 # this function, and just call forward.
   1781 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1782         or _global_backward_pre_hooks or _global_backward_hooks
   1783         or _global_forward_hooks or _global_forward_pre_hooks):
-> [1784](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1784)     return forward_call(*args, **kwargs)
   1786 result = None
   1787 called_always_called_hooks = set()

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py:192, in Qwen3VLVisionAttention.forward(self, hidden_states, cu_seqlens, rotary_pos_emb, position_embeddings, **kwargs)
    182 def forward(
    183     self,
    184     hidden_states: torch.Tensor,
   (...)
    188     **kwargs,
    189 ) -> torch.Tensor:
    190     seq_length = hidden_states.shape[0]
    191     query_states, key_states, value_states = (
--> [192](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py:192)         self.qkv(hidden_states).reshape(seq_length, 3, self.num_heads, -1).permute(1, 0, 2, 3).unbind(0)
    193     )
    194     cos, sin = position_embeddings
    195     query_states, key_states = apply_rotary_pos_emb_vision(query_states, key_states, cos, sin)

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1773, in Module._wrapped_call_impl(self, *args, **kwargs)
   1771     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1772 else:
-> [1773](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1773)     return self._call_impl(*args, **kwargs)

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1879, in Module._call_impl(self, *args, **kwargs)
   1876     return inner()
   1878 try:
-> [1879](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1879)     return inner()
   1880 except Exception:
   1881     # run always called hooks if they have not already been run
   1882     # For now only forward hooks have the always_call option but perhaps
   1883     # this functionality should be added to full backward hooks as well.
   1884     for hook_id, hook in _global_forward_hooks.items():

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1827, in Module._call_impl.<locals>.inner()
   1824     bw_hook = BackwardHook(self, full_backward_hooks, backward_pre_hooks)
   1825     args = bw_hook.setup_input_hook(args)
-> [1827](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/torch/nn/modules/module.py:1827) result = forward_call(*args, **kwargs)
   1828 if _global_forward_hooks or self._forward_hooks:
   1829     for hook_id, hook in (
   1830         *_global_forward_hooks.items(),
   1831         *self._forward_hooks.items(),
   1832     ):
   1833         # mark that always called hook is run

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/peft/tuners/lora/layer.py:745, in Linear.forward(self, x, *args, **kwargs)
    744 def forward(self, x: torch.Tensor, *args: Any, **kwargs: Any) -> torch.Tensor:
--> [745](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/peft/tuners/lora/layer.py:745)     self._check_forward_args(x, *args, **kwargs)
    746     adapter_names = kwargs.pop("adapter_names", None)
    748     if self.disable_adapters:

File /yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/peft/tuners/lora/layer.py:528, in LoraLayer._check_forward_args(self, x, *args, **kwargs)
    523 if len(x) != len(adapter_names):
    524     msg = (
    525         "Length of `adapter_names` should be the same as the number of inputs, but got "
    526         f"{len(adapter_names)} and {len(x)} respectively."
    527     )
--> [528](https://vscode-remote+ssh-002dremote-002b10-002e0-002e7-002e3.vscode-resource.vscode-cdn.net/yinghepool/zhangshuheng/miniconda3/envs/ym-qwen3vl/lib/python3.10/site-packages/peft/tuners/lora/layer.py:528)     raise ValueError(msg)
    530 if self.merged:
    531     # It is unclear what would be the right thing to do if users pass adapter_names and there are merged
    532     # adapters. Therefore, it is better to raise an error in this case.
    533     msg = "Cannot pass `adapter_names` when there are merged adapters, please call `unmerge_adapter` first."

ValueError: Length of `adapter_names` should be the same as the number of inputs, but got 1 and 18432 respectively.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing Qwen3VL inference with adapter_request and inspect the traceback through swift/llm/infer/infer_engine/pt_engine.py and swift/llm/template/base.py. Compare this path with inference without adapter_request, focusing on the vision inputs passed into generation. Done means adapter_request inference completes successfully for Qwen3VL.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.