Using Smooth Quant to perform FP8 quantization corrupts the model

I have been using a modified script based on LMDeploy to perform AWQ quantization, the main change is to add image embedding features as part of the calibration data. And this part of the code has always been working well. Since I recently planned to switch to H20 GPU for inference service deployment, I wanted to try to quantize my model to W8A8. I used the same calibration script as AWQ to execute Smooth Quant, but surprisingly, FP8 quantization broken my model. The FP8 model's output like:
`
{'id': '1', 'object': 'chat.completion', 'created': 1742808147, 'model': 'Qwen2-VL-72B', 'choices': [{'index': 0, 'message': {'role': 'assistant', 'content': '!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!', 'tool_calls': None}, 'logprobs': None, 'finish_reason': 'length'}], 'usage': {'prompt_tokens': 1544, 'total_tokens': 1576, 'completion_tokens': 32}}
`
But if int8 quantization is used, the model looks normal:
`
{'id': '1', 'object': 'chat.completion', 'created': 1742808270, 'model': 'Qwen2-VL-72B', 'choices': [{'index': 0, 'message': {'role': 'assistant', 'content': '(1)$AC = 2DE$；(2)$DE$是$\\odot O$的切线。', 'tool_calls': None}, 'logprobs': None, 'finish_reason': 'stop'}], 'usage': {'prompt_tokens': 1544, 'total_tokens': 1569, 'completion_tokens': 25}}
`