H100 multi-GPU vllm_worker startup error

Hello, when I use multiple GPUs to start inference servers on the H100 machine, an error will be reported. This is normal when using one H100 GPU. Please help me find out what the problem is. Thank you.
2 H100 failed
1 H100 success
2 A100 success
1 A100 success

Below is my environment information：
NVIDIA-SMI 525.147.05 Driver Version: 525.147.05 CUDA Version: 12.1
Python 3.10.6
torch 2.1.1
transformers 4.36.1
torch 2.1.1
vllm 0.2.5
fschat 0.2.34

The following is the error message:
INFO 12-15 00:44:33 llm_engine.py:72] Initializing an LLM engine with config: model='/meta/', tokenizer='/meta/', tokenizer_mode=slow, revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=16000, download_dir=None, load_format=auto, tensor_parallel_size=2, quantization=None, seed=0) WARNING 12-15 00:44:33 tokenizer.py:64] Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead. 2023-12-15 00:46:02 | ERROR | stderr | Traceback (most recent call last): 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2023-12-15 00:46:02 | ERROR | stderr | return _run_code(code, main_globals, None, 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2023-12-15 00:46:02 | ERROR | stderr | exec(code, run_globals) 2023-12-15 00:46:02 | ERROR | stderr | File "/code/fastchat/serve/vllm_worker.py", line 214, in <module> 2023-12-15 00:46:02 | ERROR | stderr | engine = AsyncLLMEngine.from_engine_args(engine_args) 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 486, in from_engine_args 2023-12-15 00:46:02 | ERROR | stderr | engine = cls(engine_args.worker_use_ray, 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 270, in __init__ 2023-12-15 00:46:02 | ERROR | stderr | self.engine = self._init_engine(*args, **kwargs) 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 306, in _init_engine 2023-12-15 00:46:02 | ERROR | stderr | return engine_class(*args, **kwargs) 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 111, in __init__ 2023-12-15 00:46:02 | ERROR | stderr | self._init_cache() 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 191, in _init_cache 2023-12-15 00:46:02 | ERROR | stderr | num_blocks = self._run_workers( 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 696, in _run_workers 2023-12-15 00:46:02 | ERROR | stderr | all_outputs = ray.get(all_outputs) 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/ray/_private/auto_init_hook.py", line 24, in auto_init_wrapper 2023-12-15 00:46:02 | ERROR | stderr | return fn(*args, **kwargs) 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/ray/_private/client_mode_hook.py", line 103, in wrapper 2023-12-15 00:46:02 | ERROR | stderr | return func(*args, **kwargs) 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 2563, in get 2023-12-15 00:46:02 | ERROR | stderr | raise value.as_instanceof_cause() 2023-12-15 00:46:02 | ERROR | stderr | ray.exceptions.RayTaskError(RuntimeError): ray::RayWorker.execute_method() (pid=9220, ip=10.233.119.198, actor_id=e2e7f5561e38db2f03e3973101000000, repr=<vllm.engine.ray_utils.RayWorker object at 0x7fb63eb18bb0>) 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/vllm/engine/ray_utils.py", line 32, in execute_method 2023-12-15 00:46:02 | ERROR | stderr | return executor(*args, **kwargs) 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context 2023-12-15 00:46:02 | ERROR | stderr | return func(*args, **kwargs) 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 109, in profile_num_available_blocks 2023-12-15 00:46:02 | ERROR | stderr | self.model( 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl 2023-12-15 00:46:02 | ERROR | stderr | return self._call_impl(*args, **kwargs) 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl 2023-12-15 00:46:02 | ERROR | stderr | return forward_call(*args, **kwargs) 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llama.py", line 297, in forward 2023-12-15 00:46:02 | ERROR | stderr | hidden_states = self.model(input_ids, positions, kv_caches, 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl 2023-12-15 00:46:02 | ERROR | stderr | return self._call_impl(*args, **kwargs) 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl 2023-12-15 00:46:02 | ERROR | stderr | return forward_call(*args, **kwargs) 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llama.py", line 250, in forward 2023-12-15 00:46:02 | ERROR | stderr | hidden_states = self.embed_tokens(input_ids) 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl 2023-12-15 00:46:02 | ERROR | stderr | return self._call_impl(*args, **kwargs) 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl 2023-12-15 00:46:02 | ERROR | stderr | return forward_call(*args, **kwargs) 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/parallel_utils/tensor_parallel/layers.py", line 136, in forward 2023-12-15 00:46:02 | ERROR | stderr | output = reduce_from_tensor_model_parallel_region(output_parallel) 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/parallel_utils/tensor_parallel/mappings.py", line 260, in reduce_from_tensor_model_parallel_region 2023-12-15 00:46:02 | ERROR | stderr | return _ReduceFromModelParallelRegion.apply(input_) 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/torch/autograd/function.py", line 539, in apply 2023-12-15 00:46:02 | ERROR | stderr | return super().apply(*args, **kwargs) # type: ignore[misc] 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/parallel_utils/tensor_parallel/mappings.py", line 154, in forward 2023-12-15 00:46:02 | ERROR | stderr | return _reduce(input_) 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/parallel_utils/tensor_parallel/mappings.py", line 23, in _reduce 2023-12-15 00:46:02 | ERROR | stderr | torch.distributed.all_reduce(input_, group=get_tensor_model_parallel_group()) 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 47, in wrapper 2023-12-15 00:46:02 | ERROR | stderr | return func(*args, **kwargs) 2023-12-15 00:46:02 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 2050, in all_reduce 2023-12-15 00:46:02 | ERROR | stderr | work = group.allreduce([tensor], opts) 2023-12-15 00:46:02 | ERROR | stderr | RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed.You can make a clone to get a normal tensor before doing inplace update.See https://github.com/pytorch/rfcs/pull/17 for more details. (base)

Start with fastchat/serve/vllm_worker.py at the AsyncLLMEngine initialization and compare its multi-GPU path with the reported vLLM traceback, especially the tensor-parallel all_reduce call. Reproduce the startup with two H100 GPUs using the listed Python, PyTorch, vLLM, and driver versions. Done means the two-GPU server starts without the reported inference-tensor error while the existing one-GPU and A100 cases remain working.