The discrepancy between the cosine similarity of the intermediate vectors in the VIT engine of Qwen3-8B-VL and the PyTorch-based visual part leads to a decrease in accuracy.

Describe the bug

We performed quantization on the Qwen-8B-Vl model in Drive_Orin, exported ONNX, and then used it in our engine. Related code:

tensorrt-edgellm-quantize llm \
  --model_dir /path/Qwen3-VL-8B \
  --quantization int4_awq \
  --output_dir /path

tensorrt-edgellm-export  \
     /path/Qwen3-VL-8B  \
     /path/Qwen3-VL-8B_ONNX

llm_build  \
  --onnxDir   /path/Qwen3-VL-8B_ONNX/llm  \
  --engineDir    /path/Qwen3-VL-8B_ONNX/engine/llm \
 --maxBatchSize 1 \
  --maxInputLen 10240 \
  --maxKVCacheCapacity 20480

visual_build  \
  --onnxDir    /path/Qwen3-VL-8B_ONNX/visual \
  --engineDir    /path/Qwen3-VL-8B_ONNX/engine \
  --maxImageTokens 10240 \
  --maxImageTokensPerImage 1024

 

Based on this, we conducted accuracy tests on the MMMU and MMLU datasets respectively, and the comparison is as follows:
Official MMMU dataset accuracy: https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct。
For testing tutorials, please refer to: https://github.com/NVIDIA/TensorRT-Edge-LLM/blob/main/examples/accuracy/README.md

Ours：

python3 scripts/calculate_correctness.py \
>   --predictions_file /ota/inputs_data/fkh/llm_infer_090/examples/accuracy/Outputs/mmmu_dataset.json \
>   --answers_file  /ota/inputs_data/fkh/llm_infer_090/examples/accuracy/datasets/mmmu_output/mmmu_dataset.json 
Correctness Results:
Overall Accuracy: 0.5422 (54.22%) - 488/900 correct
No subject information found in the data.

Official MMLU dataset accuracy: https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct

Ours:

python3 scripts/calculate_correctness.py \
>   --predictions_file /ota/inputs_data/fkh/llm_infer_090/examples/accuracy/Outputs/mmlu_predictions.json \
>   --answers_file  /ota/inputs_data/fkh/llm_infer_090/examples/accuracy/datasets/mmlu_output/mmlu_dataset.json
Correctness Results:
Overall Accuracy: 0.7176 (71.76%) - 10077/14042 correct
Subject-Specific Accuracy:
high_school_microeconomics: 0.9202 (92.02%) - 219/238 correct
high_school_geography: 0.8990 (89.90%) - 178/198 correct
marketing: 0.8974 (89.74%) - 210/234 correct
high_school_psychology: 0.8972 (89.72%) - 489/545 correct
high_school_government_and_politics: 0.8912 (89.12%) - 172/193 correct
high_school_biology: 0.8774 (87.74%) - 272/310 correct
management: 0.8738 (87.38%) - 90/103 correct
high_school_computer_science: 0.8700 (87.00%) - 87/100 correct
us_foreign_policy: 0.8700 (87.00%) - 87/100 correct
high_school_world_history: 0.8650 (86.50%) - 205/237 correct
astronomy: 0.8618 (86.18%) - 131/152 correct
miscellaneous: 0.8544 (85.44%) - 669/783 correct
high_school_us_history: 0.8431 (84.31%) - 172/204 correct
high_school_european_history: 0.8424 (84.24%) - 139/165 correct
sociology: 0.8408 (84.08%) - 169/201 correct
conceptual_physics: 0.8340 (83.40%) - 196/235 correct
prehistory: 0.8241 (82.41%) - 267/324 correct
high_school_macroeconomics: 0.8231 (82.31%) - 321/390 correct
logical_fallacies: 0.8221 (82.21%) - 134/163 correct
medical_genetics: 0.8200 (82.00%) - 82/100 correct
professional_medicine: 0.8125 (81.25%) - 221/272 correct
world_religions: 0.8012 (80.12%) - 137/171 correct
college_biology: 0.7847 (78.47%) - 113/144 correct
clinical_knowledge: 0.7811 (78.11%) - 207/265 correct
computer_security: 0.7800 (78.00%) - 78/100 correct
jurisprudence: 0.7593 (75.93%) - 82/108 correct
nutrition: 0.7582 (75.82%) - 232/306 correct
college_medicine: 0.7572 (75.72%) - 131/173 correct
philosophy: 0.7556 (75.56%) - 235/311 correct
electrical_engineering: 0.7379 (73.79%) - 107/145 correct
professional_psychology: 0.7369 (73.69%) - 451/612 correct
high_school_statistics: 0.7361 (73.61%) - 159/216 correct
international_law: 0.7355 (73.55%) - 89/121 correct
human_sexuality: 0.7328 (73.28%) - 96/131 correct
security_studies: 0.7306 (73.06%) - 179/245 correct
moral_disputes: 0.7283 (72.83%) - 252/346 correct
elementary_mathematics: 0.7275 (72.75%) - 275/378 correct
high_school_chemistry: 0.7192 (71.92%) - 146/203 correct
econometrics: 0.7105 (71.05%) - 81/114 correct
public_relations: 0.6909 (69.09%) - 76/110 correct
anatomy: 0.6815 (68.15%) - 92/135 correct
human_aging: 0.6771 (67.71%) - 151/223 correct
college_computer_science: 0.6700 (67.00%) - 67/100 correct
high_school_physics: 0.6358 (63.58%) - 96/151 correct
formal_logic: 0.6111 (61.11%) - 77/126 correct
college_mathematics: 0.5800 (58.00%) - 58/100 correct
professional_accounting: 0.5745 (57.45%) - 162/282 correct
college_physics: 0.5686 (56.86%) - 58/102 correct
machine_learning: 0.5625 (56.25%) - 63/112 correct
abstract_algebra: 0.5600 (56.00%) - 56/100 correct
college_chemistry: 0.5400 (54.00%) - 54/100 correct
business_ethics: 0.5300 (53.00%) - 53/100 correct
virology: 0.5241 (52.41%) - 87/166 correct
moral_scenarios: 0.4972 (49.72%) - 445/895 correct
professional_law: 0.4928 (49.28%) - 756/1534 correct
high_school_mathematics: 0.4667 (46.67%) - 126/270 correct
global_facts: 0.4000 (40.00%) - 40/100 correc

The MMMU dataset decreased by 15% and the MMLU dataset decreased by 10%. Is this a normal quantization loss?
Next, we tested the original Qwen3-8B-VL model and the Qwen3-8B-VL fp16 engine separately, and found that accuracy still decreased. We performed inference with the Qwen3-8B-VL int4 engine and inference with the Qwen3-8B-VL fp16 engine using PyTorch, and conducted layer-by-layer tests such as cosine similarity, as follows:

========== VISUAL ==========
visual_layer_0: cos=0.99991972 token_mean=0.99991635 token_min=0.99968449 max_abs=1.562500e-01 mean_abs=4.881262e-03
visual_layer_1: cos=0.99937086 token_mean=0.99930753 token_min=0.99786942 max_abs=1.691895e-01 mean_abs=8.911685e-03
visual_layer_2: cos=0.99861384 token_mean=0.99846880 token_min=0.99267314 max_abs=2.285156e-01 mean_abs=1.251065e-02
visual_layer_3: cos=0.99808005 token_mean=0.99784035 token_min=0.99011028 max_abs=3.222656e-01 mean_abs=1.398149e-02
visual_layer_4: cos=0.99788961 token_mean=0.99767568 token_min=0.98363718 max_abs=1.015625e+00 mean_abs=1.484995e-02
visual_layer_5: cos=0.99747503 token_mean=0.99714695 token_min=0.96572288 max_abs=1.328125e+00 mean_abs=1.576406e-02
visual_layer_6: cos=0.99716137 token_mean=0.99697965 token_min=0.89641439 max_abs=2.158203e+00 mean_abs=1.658124e-02
visual_layer_7: cos=0.99675094 token_mean=0.99661741 token_min=0.82914164 max_abs=3.311523e+00 mean_abs=1.770175e-02
visual_layer_8: cos=0.99638071 token_mean=0.99605696 token_min=0.73426934 max_abs=4.009766e+00 mean_abs=1.768993e-02
visual_layer_9: cos=0.98376575 token_mean=0.99451274 token_min=0.67537390 max_abs=2.238750e+02 mean_abs=2.005303e-02
visual_layer_10: cos=0.98374075 token_mean=0.99269745 token_min=0.54072691 max_abs=2.238750e+02 mean_abs=2.170384e-02
visual_layer_11: cos=0.98394836 token_mean=0.99097673 token_min=0.44190450 max_abs=2.241250e+02 mean_abs=2.502824e-02
visual_layer_12: cos=0.98372403 token_mean=0.98728279 token_min=0.24847147 max_abs=2.240000e+02 mean_abs=3.021091e-02
visual_layer_13: cos=0.98398022 token_mean=0.98439266 token_min=0.15625883 max_abs=2.240000e+02 mean_abs=3.632143e-02
visual_layer_14: cos=0.98395618 token_mean=0.98273023 token_min=0.08135351 max_abs=2.237500e+02 mean_abs=4.367605e-02
visual_layer_15: cos=0.98399265 token_mean=0.98138744 token_min=0.10133473 max_abs=2.235000e+02 mean_abs=5.100240e-02
visual_layer_16: cos=0.98435183 token_mean=0.98158979 token_min=0.17911310 max_abs=2.235000e+02 mean_abs=5.915902e-02
visual_layer_17: cos=0.98371173 token_mean=0.97886274 token_min=0.11530280 max_abs=2.235000e+02 mean_abs=6.918409e-02
visual_layer_18: cos=0.98341023 token_mean=0.97657096 token_min=0.11966794 max_abs=2.235000e+02 mean_abs=8.231049e-02
visual_layer_19: cos=0.98414962 token_mean=0.97978606 token_min=0.24113462 max_abs=2.235000e+02 mean_abs=9.713007e-02
visual_layer_20: cos=0.98538561 token_mean=0.98183546 token_min=0.33449753 max_abs=2.260000e+02 mean_abs=1.079775e-01
visual_layer_21: cos=0.98517046 token_mean=0.98092751 token_min=0.32084668 max_abs=2.270000e+02 mean_abs=1.214576e-01
visual_layer_22: cos=0.98889349 token_mean=0.98727807 token_min=0.54777152 max_abs=2.275000e+02 mean_abs=1.336807e-01
visual_layer_23: cos=0.98635645 token_mean=0.98358329 token_min=0.45137762 max_abs=2.285000e+02 mean_abs=1.545347e-01
visual_layer_24: cos=0.98987077 token_mean=0.98874478 token_min=0.57996301 max_abs=2.275000e+02 mean_abs=1.848451e-01
visual_layer_25: cos=0.97477325 token_mean=0.96705706 token_min=0.08465578 max_abs=2.275000e+02 mean_abs=2.565090e-01
visual_layer_26: cos=0.95338153 token_mean=0.99673902 token_min=0.70639956 max_abs=1.443475e+04 mean_abs=4.391930e+00
========== LLM PREFILL ==========
llm_prefill_layer_0: cos=0.96257749 token_mean=0.96608830 token_min=0.16761210 max_abs=6.287598e+00 mean_abs=6.973393e-02
llm_prefill_layer_1: cos=0.97317458 token_mean=0.96986693 token_min=0.22177679 max_abs=1.542188e+01 mean_abs=7.897930e-02
llm_prefill_layer_2: cos=0.97497147 token_mean=0.97096219 token_min=0.26280901 max_abs=1.839844e+01 mean_abs=8.494971e-02
llm_prefill_layer_3: cos=0.97532984 token_mean=0.97079387 token_min=0.24206663 max_abs=2.310742e+01 mean_abs=8.828703e-02
llm_prefill_layer_4: cos=0.97858126 token_mean=0.97559189 token_min=0.30830709 max_abs=2.576562e+01 mean_abs=9.388752e-02
llm_prefill_layer_5: cos=0.98142462 token_mean=0.97863878 token_min=0.35684857 max_abs=1.945312e+01 mean_abs=9.797349e-02
llm_prefill_layer_6: cos=0.99940536 token_mean=0.98029180 token_min=0.43764516 max_abs=1.360000e+02 mean_abs=1.042746e-01
llm_prefill_layer_7: cos=0.99935064 token_mean=0.97683343 token_min=0.39926915 max_abs=1.360000e+02 mean_abs=1.121275e-01
llm_prefill_layer_8: cos=0.99929767 token_mean=0.97364760 token_min=0.36835602 max_abs=1.360000e+02 mean_abs=1.207454e-01
llm_prefill_layer_9: cos=0.99924989 token_mean=0.97024911 token_min=0.31823740 max_abs=1.360000e+02 mean_abs=1.256017e-01
llm_prefill_layer_10: cos=0.99918384 token_mean=0.96809741 token_min=0.27467920 max_abs=1.360000e+02 mean_abs=1.306931e-01
llm_prefill_layer_11: cos=0.99907784 token_mean=0.96554639 token_min=0.24464166 max_abs=1.360000e+02 mean_abs=1.404774e-01
llm_prefill_layer_12: cos=0.99898031 token_mean=0.96586801 token_min=0.30326568 max_abs=1.360000e+02 mean_abs=1.500715e-01
llm_prefill_layer_13: cos=0.99882930 token_mean=0.96363824 token_min=0.28421547 max_abs=1.360000e+02 mean_abs=1.643562e-01
llm_prefill_layer_14: cos=0.99873303 token_mean=0.96251513 token_min=0.31065852 max_abs=1.360000e+02 mean_abs=1.736298e-01
llm_prefill_layer_15: cos=0.99848576 token_mean=0.95893112 token_min=0.33217780 max_abs=1.360000e+02 mean_abs=1.941364e-01
llm_prefill_layer_16: cos=0.99815631 token_mean=0.95523078 token_min=0.35387726 max_abs=3.760000e+02 mean_abs=2.075825e-01
llm_prefill_layer_17: cos=0.99775325 token_mean=0.94830762 token_min=0.39107840 max_abs=3.700000e+02 mean_abs=2.405873e-01
llm_prefill_layer_18: cos=0.99717206 token_mean=0.94523345 token_min=0.40801401 max_abs=3.700000e+02 mean_abs=2.814683e-01
llm_prefill_layer_19: cos=0.99640579 token_mean=0.94390971 token_min=0.43211081 max_abs=3.700000e+02 mean_abs=3.266302e-01
llm_prefill_layer_20: cos=0.99541878 token_mean=0.93411558 token_min=0.40604693 max_abs=3.680000e+02 mean_abs=3.767530e-01
llm_prefill_layer_21: cos=0.99397069 token_mean=0.92584189 token_min=0.37347191 max_abs=3.680000e+02 mean_abs=4.398039e-01
llm_prefill_layer_22: cos=0.99236213 token_mean=0.92055113 token_min=0.34011349 max_abs=3.680000e+02 mean_abs=4.995144e-01
llm_prefill_layer_23: cos=0.98866897 token_mean=0.91558554 token_min=0.32029853 max_abs=3.660000e+02 mean_abs=6.240554e-01
llm_prefill_layer_24: cos=0.98317569 token_mean=0.91326579 token_min=0.29063660 max_abs=3.660000e+02 mean_abs=7.806778e-01
llm_prefill_layer_25: cos=0.97953589 token_mean=0.90826914 token_min=0.28580168 max_abs=3.660000e+02 mean_abs=8.917436e-01
llm_prefill_layer_26: cos=0.97454767 token_mean=0.90397072 token_min=0.34308243 max_abs=3.660000e+02 mean_abs=1.032910e+00
llm_prefill_layer_27: cos=0.96882117 token_mean=0.89548758 token_min=0.31699387 max_abs=3.640000e+02 mean_abs=1.186214e+00
llm_prefill_layer_28: cos=0.96116547 token_mean=0.88512434 token_min=0.27929208 max_abs=3.620000e+02 mean_abs=1.387045e+00
llm_prefill_layer_29: cos=0.95363240 token_mean=0.88109387 token_min=0.28447398 max_abs=3.600000e+02 mean_abs=1.625224e+00
llm_prefill_layer_30: cos=0.94632333 token_mean=0.87770726 token_min=0.27738044 max_abs=3.580000e+02 mean_abs=1.907698e+00
llm_prefill_layer_31: cos=0.93827115 token_mean=0.87665022 token_min=0.26296617 max_abs=3.580000e+02 mean_abs=2.238904e+00
llm_prefill_layer_32: cos=0.92907938 token_mean=0.86963134 token_min=0.23480066 max_abs=3.560000e+02 mean_abs=2.603658e+00
llm_prefill_layer_33: cos=0.92368615 token_mean=0.86371021 token_min=0.16171548 max_abs=3.540000e+02 mean_abs=2.992549e+00
llm_prefill_layer_34: cos=0.92999846 token_mean=0.88497315 token_min=0.29039692 max_abs=7.100000e+02 mean_abs=3.573124e+00
llm_prefill_layer_35: cos=0.82408727 token_mean=0.78291030 token_min=-0.01085496 max_abs=1.112000e+03 mean_abs=4.709686e+00
========== LLM DECODE STEP 0 ==========
llm_decode_step_0_layer_0: cos=0.99606514 token_mean=0.99606514 token_min=0.99606514 max_abs=1.015625e-01 mean_abs=1.288980e-02
llm_decode_step_0_layer_1: cos=0.99639024 token_mean=0.99639024 token_min=0.99639024 max_abs=2.695312e-01 mean_abs=1.478732e-02
llm_decode_step_0_layer_2: cos=0.99663143 token_mean=0.99663143 token_min=0.99663143 max_abs=3.437500e-01 mean_abs=1.706620e-02
llm_decode_step_0_layer_3: cos=0.99613404 token_mean=0.99613404 token_min=0.99613404 max_abs=2.968750e-01 mean_abs=2.252953e-02
llm_decode_step_0_layer_4: cos=0.99530324 token_mean=0.99530324 token_min=0.99530324 max_abs=2.421875e-01 mean_abs=3.391664e-02
llm_decode_step_0_layer_5: cos=0.99529633 token_mean=0.99529633 token_min=0.99529633 max_abs=7.500000e-01 mean_abs=4.534728e-02
llm_decode_step_0_layer_6: cos=0.99505362 token_mean=0.99505362 token_min=0.99505362 max_abs=8.281250e-01 mean_abs=5.239297e-02
llm_decode_step_0_layer_7: cos=0.99446000 token_mean=0.99446000 token_min=0.99446000 max_abs=1.078125e+00 mean_abs=5.694513e-02
llm_decode_step_0_layer_8: cos=0.99322710 token_mean=0.99322710 token_min=0.99322710 max_abs=4.404297e-01 mean_abs=6.655497e-02
llm_decode_step_0_layer_9: cos=0.99240469 token_mean=0.99240469 token_min=0.99240469 max_abs=4.586182e-01 mean_abs=7.316023e-02
llm_decode_step_0_layer_10: cos=0.99242517 token_mean=0.99242517 token_min=0.99242517 max_abs=4.785156e-01 mean_abs=7.574357e-02
llm_decode_step_0_layer_11: cos=0.99135425 token_mean=0.99135425 token_min=0.99135425 max_abs=5.937500e-01 mean_abs=8.182604e-02
llm_decode_step_0_layer_12: cos=0.99088546 token_mean=0.99088546 token_min=0.99088546 max_abs=6.728516e-01 mean_abs=8.841901e-02
llm_decode_step_0_layer_13: cos=0.98964204 token_mean=0.98964204 token_min=0.98964204 max_abs=7.236328e-01 mean_abs=9.985789e-02
llm_decode_step_0_layer_14: cos=0.98983790 token_mean=0.98983790 token_min=0.98983790 max_abs=6.250000e-01 mean_abs=1.003425e-01
llm_decode_step_0_layer_15: cos=0.98962065 token_mean=0.98962065 token_min=0.98962065 max_abs=8.134766e-01 mean_abs=1.072624e-01
llm_decode_step_0_layer_16: cos=0.98864032 token_mean=0.98864032 token_min=0.98864032 max_abs=7.539062e-01 mean_abs=1.134188e-01
llm_decode_step_0_layer_17: cos=0.98798696 token_mean=0.98798696 token_min=0.98798696 max_abs=9.218750e-01 mean_abs=1.264119e-01
llm_decode_step_0_layer_18: cos=0.98804802 token_mean=0.98804802 token_min=0.98804802 max_abs=1.625000e+00 mean_abs=1.476751e-01
llm_decode_step_0_layer_19: cos=0.98463882 token_mean=0.98463882 token_min=0.98463882 max_abs=1.222656e+00 mean_abs=1.844821e-01
llm_decode_step_0_layer_20: cos=0.98218886 token_mean=0.98218886 token_min=0.98218886 max_abs=1.324219e+00 mean_abs=2.197816e-01
llm_decode_step_0_layer_21: cos=0.97853291 token_mean=0.97853291 token_min=0.97853291 max_abs=2.562500e+00 mean_abs=2.886499e-01
llm_decode_step_0_layer_22: cos=0.97588249 token_mean=0.97588249 token_min=0.97588249 max_abs=3.500000e+00 mean_abs=3.265775e-01
llm_decode_step_0_layer_23: cos=0.97333422 token_mean=0.97333422 token_min=0.97333422 max_abs=4.875000e+00 mean_abs=4.099910e-01
llm_decode_step_0_layer_24: cos=0.97530047 token_mean=0.97530047 token_min=0.97530047 max_abs=3.625000e+00 mean_abs=4.949793e-01
llm_decode_step_0_layer_25: cos=0.97541489 token_mean=0.97541489 token_min=0.97541489 max_abs=6.500000e+00 mean_abs=5.707377e-01
llm_decode_step_0_layer_26: cos=0.97740827 token_mean=0.97740827 token_min=0.97740827 max_abs=6.687500e+00 mean_abs=6.442490e-01
llm_decode_step_0_layer_27: cos=0.97629688 token_mean=0.97629688 token_min=0.97629688 max_abs=1.031250e+01 mean_abs=7.230831e-01
llm_decode_step_0_layer_28: cos=0.97824682 token_mean=0.97824682 token_min=0.97824682 max_abs=1.100000e+01 mean_abs=8.320602e-01
llm_decode_step_0_layer_29: cos=0.97988986 token_mean=0.97988986 token_min=0.97988986 max_abs=1.425000e+01 mean_abs=9.403001e-01
llm_decode_step_0_layer_30: cos=0.98225391 token_mean=0.98225391 token_min=0.98225391 max_abs=1.450000e+01 mean_abs=1.088904e+00
llm_decode_step_0_layer_31: cos=0.98516518 token_mean=0.98516518 token_min=0.98516518 max_abs=1.725000e+01 mean_abs=1.224947e+00
llm_decode_step_0_layer_32: cos=0.98679413 token_mean=0.98679413 token_min=0.98679413 max_abs=2.775000e+01 mean_abs=1.358807e+00
llm_decode_step_0_layer_33: cos=0.98743695 token_mean=0.98743695 token_min=0.98743695 max_abs=4.250000e+01 mean_abs=1.785630e+00
llm_decode_step_0_layer_34: cos=0.99164206 token_mean=0.99164206 token_min=0.99164206 max_abs=4.050000e+01 mean_abs=2.049278e+00
llm_decode_step_0_layer_35: cos=0.93379602 token_mean=0.93379602 token_min=0.93379602 max_abs=7.350000e+01 mean_abs=2.378285e+00

After analysis, we believe that the anomaly is in the 9th and 26th layers of the visual engine, which leads to a decrease in subsequent accuracy. Are there any solutions for this?

Environment

• Platform: NVIDIA DRIVE Orin

• DRIVE OS: 6.0.12.1

• CUDA: 11.4.30

• cuDNN: 8.9.2

• TensorRT-10.13.0.31

• Ubuntu: 20.04.6 LTS

• Kernel: 5.15.163-rt-tegra

• Platform: NVIDIA DRIVE Thor

• DRIVE OS: 7.0.5.0

• Ubuntu: 24.04

• Linux Kernel: 6.1

• GCC: 13.2

• CUDA: 13.0.1

• cuDNN: 9.13.0

• TensorRT: 10.14.2

Start with examples/accuracy/README.md and scripts/calculate_correctness.py to reproduce the MMMU and MMLU measurements using the reported build and inference commands. Compare the visual and LLM prefill layer-by-layer cosine results to locate where the engine diverges from PyTorch. Done means identifying the cause of the discrepancy and restoring accuracy, with evidence from the accuracy scripts and intermediate comparisons.