Qwen2-Audio-7B-Instruct无法正常推理

Checklist / 检查清单

• I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues，确认这是一个新的 bug report。

Bug Description / Bug 描述

使用未训练的模型进行推理，传入wav文件，无法输出正确的转录结果。

xx@szaic-hpc-debug-0003:/workspace$ bash scripts/decode_qwenaudio.sh
/usr/local/python3.10.15/lib/python3.10/site-packages/torch_npu/utils/collect_env.py:58: UserWarning: Warning: The /usr/local/Ascend/ascend-toolkit/latest owner does not match the current owner.
  warnings.warn(f"Warning: The {path} owner does not match the current owner.")
/usr/local/python3.10.15/lib/python3.10/site-packages/torch_npu/utils/collect_env.py:58: UserWarning: Warning: The /usr/local/Ascend/ascend-toolkit/8.2.RC1/aarch64-linux/ascend_toolkit_install.info owner does not match the current owner.
  warnings.warn(f"Warning: The {path} owner does not match the current owner.")
/usr/local/python3.10.15/lib/python3.10/site-packages/torch_npu/utils/_path_manager.py:66: UserWarning: Permission mismatch: The owner of /usr/local/python3.10.15/lib/python3.10/site-packages/torch_npu/lib/libop_plugin_atb.so does not match.
  warnings.warn(f"Permission mismatch: The owner of {path} does not match.")
/usr/local/python3.10.15/lib/python3.10/site-packages/torch_npu/utils/collect_env.py:58: UserWarning: Warning: The /usr/local/Ascend/ascend-toolkit/latest owner does not match the current owner.
  warnings.warn(f"Warning: The {path} owner does not match the current owner.")
/usr/local/python3.10.15/lib/python3.10/site-packages/torch_npu/utils/collect_env.py:58: UserWarning: Warning: The /usr/local/Ascend/ascend-toolkit/8.2.RC1/aarch64-linux/ascend_toolkit_install.info owner does not match the current owner.
  warnings.warn(f"Warning: The {path} owner does not match the current owner.")
/usr/local/python3.10.15/lib/python3.10/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: '/usr/local/python3.10.15/lib/python3.10/site-packages/torchvision/image.so: undefined symbol: _ZN3c1017RegisterOperatorsD1Ev'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
  warn(
run sh: `/usr/local/python3.10.15/bin/python3.10 /usr/local/python3.10.15/lib/python3.10/site-packages/swift/cli/infer.py --model /models/Qwen/Qwen2-Audio-7B-Instruct --infer_backend transformers --stream true --temperature 0 --max_new_tokens 512 --max_batch_size 1`
/usr/local/python3.10.15/lib/python3.10/site-packages/torch_npu/utils/collect_env.py:58: UserWarning: Warning: The /usr/local/Ascend/ascend-toolkit/latest owner does not match the current owner.
  warnings.warn(f"Warning: The {path} owner does not match the current owner.")
/usr/local/python3.10.15/lib/python3.10/site-packages/torch_npu/utils/collect_env.py:58: UserWarning: Warning: The /usr/local/Ascend/ascend-toolkit/8.2.RC1/aarch64-linux/ascend_toolkit_install.info owner does not match the current owner.
  warnings.warn(f"Warning: The {path} owner does not match the current owner.")
/usr/local/python3.10.15/lib/python3.10/site-packages/torch_npu/utils/_path_manager.py:66: UserWarning: Permission mismatch: The owner of /usr/local/python3.10.15/lib/python3.10/site-packages/torch_npu/lib/libop_plugin_atb.so does not match.
  warnings.warn(f"Permission mismatch: The owner of {path} does not match.")
/usr/local/python3.10.15/lib/python3.10/site-packages/torch_npu/utils/collect_env.py:58: UserWarning: Warning: The /usr/local/Ascend/ascend-toolkit/latest owner does not match the current owner.
  warnings.warn(f"Warning: The {path} owner does not match the current owner.")
/usr/local/python3.10.15/lib/python3.10/site-packages/torch_npu/utils/collect_env.py:58: UserWarning: Warning: The /usr/local/Ascend/ascend-toolkit/8.2.RC1/aarch64-linux/ascend_toolkit_install.info owner does not match the current owner.
  warnings.warn(f"Warning: The {path} owner does not match the current owner.")
/usr/local/python3.10.15/lib/python3.10/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: '/usr/local/python3.10.15/lib/python3.10/site-packages/torchvision/image.so: undefined symbol: _ZN3c1017RegisterOperatorsD1Ev'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
  warn(
[INFO:swift] Set HCCL_CONNECT_TIMEOUT=600 by default for NPU.
[INFO:swift] Successfully registered `/usr/local/python3.10.15/lib/python3.10/site-packages/swift/dataset/data/dataset_info.json`.
[INFO:swift] rank: -1, local_rank: -1, world_size: 1, local_world_size: 1
`torch_dtype` is deprecated! Use `dtype` instead!
[INFO:swift] Setting torch_dtype: torch.bfloat16
[WARNING:swift] Please install the package: `pip install "transformers>=4.45,<4.49" -U`.
[INFO:swift] Setting args.eval_human: True
[INFO:swift] Setting args.lazy_tokenize: True
[INFO:swift] args: InferArguments(use_ray=False, ray_exp_name=None, device_groups=None, model='/models/Qwen/Qwen2-Audio-7B-Instruct', model_type='qwen2_audio', model_revision=None, task_type='causal_lm', torch_dtype=torch.bfloat16, attn_impl=None, experts_impl=None, new_special_tokens=[], num_labels=None, problem_type=None, rope_scaling=None, device_map=None, max_memory={}, max_model_len=None, local_repo_path=None, init_strategy=None, template='qwen2_audio', system=None, max_length=8192, truncation_strategy='delete', max_pixels=None, agent_template=None, norm_bbox=None, use_chat_template=True, padding_side='right', padding_free=False, loss_scale='default', sequence_parallel_size=1, template_backend='swift', response_prefix=None, enable_thinking=None, add_non_thinking_prefix=True, dataset=[], val_dataset=[], cached_dataset=[], cached_val_dataset=[], split_dataset_ratio=0.0, data_seed=42, dataset_num_proc=1, load_from_cache_file=False, dataset_shuffle=True, val_dataset_shuffle=False, streaming=False, interleave_prob=None, stopping_strategy='first_exhausted', shuffle_buffer_size=1000, download_mode='reuse_dataset_if_exists', columns={}, strict=False, remove_unused_columns=True, model_name=None, model_author=None, custom_dataset_info=[], quant_method=None, quant_bits=None, hqq_axis=None, bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_quant_type='nf4', bnb_4bit_use_double_quant=True, bnb_4bit_quant_storage=None, max_new_tokens=512, temperature=0.0, top_k=None, top_p=None, repetition_penalty=None, num_beams=1, stream=True, stop_words=[], logprobs=False, top_logprobs=None, structured_outputs_regex=None, tuner_backend='peft', tuner_type='lora', train_type=None, adapters=[], external_plugins=[], custom_register_path=[], seed=42, model_kwargs={}, load_args=True, load_data_args=False, packing=False, packing_length=None, packing_num_proc=1, lazy_tokenize=True, use_hf=False, hub_token=None, ddp_timeout=18000000, ddp_backend=None, ignore_args_error=False, use_swift_lora=False, vllm_gpu_memory_utilization=0.9, vllm_tensor_parallel_size=1, vllm_pipeline_parallel_size=1, vllm_enable_expert_parallel=False, vllm_max_num_seqs=256, vllm_max_model_len=None, vllm_disable_custom_all_reduce=True, vllm_enforce_eager=False, vllm_limit_mm_per_prompt=None, vllm_max_lora_rank=16, vllm_enable_prefix_caching=None, vllm_use_async_engine=False, vllm_quantization=None, vllm_reasoning_parser=None, vllm_disable_cascade_attn=False, vllm_mm_processor_cache_gb=None, vllm_speculative_config=None, vllm_engine_kwargs={}, vllm_data_parallel_size=1, sglang_tp_size=1, sglang_pp_size=1, sglang_dp_size=1, sglang_ep_size=1, sglang_enable_ep_moe=False, sglang_mem_fraction_static=None, sglang_context_length=None, sglang_disable_cuda_graph=False, sglang_quantization=None, sglang_kv_cache_dtype='auto', sglang_enable_dp_attention=False, sglang_disable_custom_all_reduce=True, sglang_speculative_algorithm=None, sglang_speculative_num_steps=None, sglang_speculative_eagle_topk=None, sglang_speculative_num_draft_tokens=None, lmdeploy_tp=1, lmdeploy_session_len=None, lmdeploy_cache_max_entry_count=0.8, lmdeploy_quant_policy=0, lmdeploy_vision_batch_size=1, merge_lora=False, safe_serialization=True, max_shard_size='5GB', infer_backend='transformers', result_path=None, write_batch_size=1000, metric=None, max_batch_size=1, val_dataset_sample=None, reranker_use_activation=True)
[INFO:swift] Global seed set to 42
[WARNING:swift] Please install the package: `pip install "transformers>=4.45,<4.49" -U`.
[INFO:swift] model_kwargs: {'device_map': 'npu:0', 'dtype': torch.bfloat16}
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:23<00:00,  4.76s/it]
[INFO:swift] default_system: 'You are a helpful assistant.'
[INFO:swift] max_length: 8192
[INFO:swift] response_prefix: ''
[INFO:swift] agent_template: hermes
[INFO:swift] norm_bbox: norm1000
[INFO:swift] Using environment variable `ROOT_IMAGE_DIR`, Setting ROOT_IMAGE_DIR: /data.
[INFO:swift] Setting sampling_rate: 16000. You can adjust this hyperparameter through the environment variable: `SAMPLING_RATE`.
[INFO:swift] model: Qwen2AudioForConditionalGeneration(
  (audio_tower): Qwen2AudioEncoder(
    (conv1): Conv1d(128, 1280, kernel_size=(3,), stride=(1,), padding=(1,))
    (conv2): Conv1d(1280, 1280, kernel_size=(3,), stride=(2,), padding=(1,))
    (embed_positions): Embedding(1500, 1280)
    (layers): ModuleList(
      (0-31): 32 x Qwen2AudioEncoderLayer(
        (self_attn): Qwen2AudioAttention(
          (k_proj): Linear(in_features=1280, out_features=1280, bias=False)
          (v_proj): Linear(in_features=1280, out_features=1280, bias=True)
          (q_proj): Linear(in_features=1280, out_features=1280, bias=True)
          (out_proj): Linear(in_features=1280, out_features=1280, bias=True)
        )
        (self_attn_layer_norm): LayerNorm((1280,), eps=1e-05, elementwise_affine=True)
        (activation_fn): GELUActivation()
        (fc1): Linear(in_features=1280, out_features=5120, bias=True)
        (fc2): Linear(in_features=5120, out_features=1280, bias=True)
        (final_layer_norm): LayerNorm((1280,), eps=1e-05, elementwise_affine=True)
      )
    )
    (layer_norm): LayerNorm((1280,), eps=1e-05, elementwise_affine=True)
    (avg_pooler): AvgPool1d(kernel_size=(2,), stride=(2,), padding=(0,))
  )
  (multi_modal_projector): Qwen2AudioMultiModalProjector(
    (linear): Linear(in_features=1280, out_features=4096, bias=True)
  )
  (language_model): Qwen2ForCausalLM(
    (model): Qwen2Model(
      (embed_tokens): Embedding(156032, 4096)
      (layers): ModuleList(
        (0-31): 32 x Qwen2DecoderLayer(
          (self_attn): Qwen2Attention(
            (q_proj): Linear(in_features=4096, out_features=4096, bias=True)
            (k_proj): Linear(in_features=4096, out_features=4096, bias=True)
            (v_proj): Linear(in_features=4096, out_features=4096, bias=True)
            (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
          )
          (mlp): Qwen2MLP(
            (gate_proj): Linear(in_features=4096, out_features=11008, bias=False)
            (up_proj): Linear(in_features=4096, out_features=11008, bias=False)
            (down_proj): Linear(in_features=11008, out_features=4096, bias=False)
            (act_fn): SiLUActivation()
          )
          (input_layernorm): NpuRMSNorm((4096,), eps=1e-05)
          (post_attention_layernorm): NpuRMSNorm((4096,), eps=1e-05)
        )
      )
      (norm): NpuRMSNorm((4096,), eps=1e-05)
      (rotary_emb): Qwen2RotaryEmbedding()
    )
    (lm_head): Linear(in_features=4096, out_features=156032, bias=False)
  )
)
[INFO:swift] Start time of running main: 2026-03-16 18:33:02.453295
[INFO:swift] swift.__version__: 4.0.2
[INFO:swift] request_config: RequestConfig(max_tokens=512, temperature=0.0, top_k=None, top_p=None, repetition_penalty=None, num_beams=1, stop=[], seed=None, stream=True, logprobs=False, top_logprobs=None, n=1, best_of=None, presence_penalty=0.0, frequency_penalty=0.0, length_penalty=1.0, return_details=False, structured_outputs_regex=None)
[INFO:swift] Input `exit` or `quit` to exit the conversation.
[INFO:swift] Input `multi-line` to switch to multi-line input mode.
[INFO:swift] Input `reset-system` to reset the system and clear the history.
[INFO:swift] Input `clear` to clear the history.
<<< nihao
你好！您好吗？
Hello! How are you?
--------------------------------------------------
<<< Transcript the following audio.<audio>
Input an audio path or URL <<< slidespeech/test_oracle_v1/wavs/9306-43670-0010.wav
Expanding inputs for audio tokens in Qwen2Audio should be done in processing.
12500utopian happy to doinquis
I'my to be here's he has happened
andrew
is coming to becom hissing
<|im_start|>
<|im_start|>
<|im_start|> his mouth for us up the show for us."
--------------------------------------------------

How to Reproduce / 如何复现

ms_swift==4.0.2
transformers==4.57.6
torch==2.6.0+cpu
torch_npu==2.6.0

ROOT_IMAGE_DIR=/data \
ASCEND_RT_VISIBLE_DEVICES=0 \
swift infer \
    --model Qwen/Qwen2-Audio-7B-Instruct \
    --infer_backend transformers \
    --stream true \
    --temperature 0 \
    --max_new_tokens 512 \

Additional Information / 补充信息

No response

Start by running scripts/decode_qwenaudio.sh and reading the swift/cli/infer.py entry point used in the report. Compare the Qwen2-Audio-7B-Instruct transcription for the wav input with the expected result, while checking the logged torch_npu, torchvision, and transformers environment warnings. Done means inference produces the correct transcription.