Error executing method determine_num_available_blocks: vLLM multi node fails for both DeepSeek-Coder-V2-Instruct and DeepSeek-Coder-V2-Lite-Instruct

首先想问一下DeepSeek有没有试过在**vLLM multi node**上运行过？
我是通过ray在2个node x 8 GPUs V100上以half(float16)运行

这是运行参数:

`CUDA_LAUNCH_BLOCKING=1 OMP_NUM_THREADS=1 vllm serve deepseek-ai/DeepSeek-Coder-V2-Instruct --tensor-parallel-size 16 --dtype half --trust-remote-code --enforce-eager --enable-chunked-prefill=False `

DeepSeek-Coder-V2-Lite-Instruct也是在**determine_num_available_blocks** 处fails, 但是报一个NCCL error:

(RayWorkerWrapper pid=23558, ip=10.0.128.18) ERROR 07-28 13:53:40 worker_base.py:382] RuntimeError: NCCL Error 3: internal error - please report this issue to the NCCL developers

```
root@g02-17:/vllm-workspace# PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CUDA_LAUNCH_BLOCKING=1 OMP_NUM_THREADS=1 HF_ENDPOINT="https://hf-mirror.com" vllm serve deepseek-ai/DeepSeek-Coder-V2-Instruct --tensor-parallel-size 16 --dtype half --trust-remote-code --enforce-eager --enable-chunked-prefill=False --max-model-len 8192
INFO 07-28 13:54:35 api_server.py:219] vLLM API server version 0.5.3.post1
INFO 07-28 13:54:35 api_server.py:220] args: Namespace(model_tag='deepseek-ai/DeepSeek-Coder-V2-Instruct', host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], model='deepseek-ai/DeepSeek-Coder-V2-Instruct', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, download_dir=None, load_format='auto', dtype='half', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=8192, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=16, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=True, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', scheduler_delay_factor=0.0, enable_chunked_prefill=False, speculative_model=None, num_speculative_tokens=None, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, engine_use_ray=False, disable_log_requests=False, max_log_len=None, dispatch_function=)
WARNING 07-28 13:54:37 config.py:1425] Casting torch.bfloat16 to torch.float16.
INFO 07-28 13:54:37 config.py:715] Defaulting to use ray for distributed inference
2024-07-28 13:54:37,131 INFO worker.py:1603 -- Connecting to existing Ray cluster at address: 10.0.128.17:6379...
2024-07-28 13:54:37,140 INFO worker.py:1788 -- Connected to Ray cluster.
INFO 07-28 13:54:38 llm_engine.py:176] Initializing an LLM engine (v0.5.3.post1) with config: model='deepseek-ai/DeepSeek-Coder-V2-Instruct', speculative_config=None, tokenizer='deepseek-ai/DeepSeek-Coder-V2-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=16, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=deepseek-ai/DeepSeek-Coder-V2-Instruct, use_v2_block_manager=False, enable_prefix_caching=False)
INFO 07-28 13:55:28 selector.py:151] Cannot use FlashAttention-2 backend for Volta and Turing GPUs.
INFO 07-28 13:55:28 selector.py:54] Using XFormers backend.
(RayWorkerWrapper pid=24600, ip=10.0.128.18) INFO 07-28 13:55:28 selector.py:151] Cannot use FlashAttention-2 backend for Volta and Turing GPUs.
(RayWorkerWrapper pid=24600, ip=10.0.128.18) INFO 07-28 13:55:28 selector.py:54] Using XFormers backend.
[W CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
INFO 07-28 13:55:32 utils.py:784] Found nccl from library libnccl.so.2
INFO 07-28 13:55:32 pynccl.py:63] vLLM is using nccl==2.20.5
(RayWorkerWrapper pid=77233) INFO 07-28 13:55:32 utils.py:784] Found nccl from library libnccl.so.2
(RayWorkerWrapper pid=77233) INFO 07-28 13:55:32 pynccl.py:63] vLLM is using nccl==2.20.5
WARNING 07-28 13:55:33 custom_all_reduce.py:69] Custom allreduce is disabled because this process group spans across nodes.
INFO 07-28 13:55:33 shm_broadcast.py:241] vLLM message queue communication handle: Handle(connect_ip='10.0.128.17', local_reader_ranks=[1, 2, 3, 4, 5, 6, 7], buffer=, local_subscribe_port=59601, local_sync_port=35331, remote_subscribe_port=53347, remote_sync_port=58569)
(RayWorkerWrapper pid=24524, ip=10.0.128.18) WARNING 07-28 13:55:33 custom_all_reduce.py:69] Custom allreduce is disabled because this process group spans across nodes.
INFO 07-28 13:55:33 model_runner.py:680] Starting to load model deepseek-ai/DeepSeek-Coder-V2-Instruct...
(RayWorkerWrapper pid=24524, ip=10.0.128.18) INFO 07-28 13:55:33 model_runner.py:680] Starting to load model deepseek-ai/DeepSeek-Coder-V2-Instruct...
(RayWorkerWrapper pid=25062, ip=10.0.128.18) INFO 07-28 13:55:28 selector.py:151] Cannot use FlashAttention-2 backend for Volta and Turing GPUs. [repeated 14x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(RayWorkerWrapper pid=25062, ip=10.0.128.18) INFO 07-28 13:55:28 selector.py:54] Using XFormers backend. [repeated 14x across cluster]
(RayWorkerWrapper pid=24524, ip=10.0.128.18) Cache shape torch.Size([163840, 64])
Cache shape torch.Size([163840, 64])
INFO 07-28 13:55:33 selector.py:151] Cannot use FlashAttention-2 backend for Volta and Turing GPUs.
INFO 07-28 13:55:33 selector.py:54] Using XFormers backend.
INFO 07-28 13:55:34 weight_utils.py:223] Using model weights format ['*.safetensors']
(RayWorkerWrapper pid=24600, ip=10.0.128.18) INFO 07-28 13:55:35 weight_utils.py:223] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards: 0% Completed | 0/55 [00:00