Multi GPU setup in vLLm

I changed the related parameters in the provided python code to run the model on top of 2 RTX 3060. But i faced with this Error:
```
python run_dpsk_ocr_pdf.py
INFO 10-22 20:23:17 [__init__.py:239] Automatically detected platform cuda.
INFO 10-22 20:23:20 [config.py:456] Overriding HF config with {'architectures': ['DeepseekOCRForCausalLM']}
INFO 10-22 20:23:20 [config.py:717] This model supports multiple tasks: {'score', 'generate', 'classify', 'reward', 'embed'}. Defaulting to 'generate'.
INFO 10-22 20:23:20 [llm_engine.py:240] Initializing a V0 LLM engine (v0.8.5) with config: model='../../DeepSeek-OCR', speculative_config=None, tokenizer='../../DeepSeek-OCR', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=../../DeepSeek-OCR, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=None, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=True, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":104}, use_cached_outputs=False,
INFO 10-22 20:23:20 [cuda.py:292] Using Flash Attention backend.
INFO 10-22 20:23:21 [parallel_state.py:1004] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0
INFO 10-22 20:23:21 [model_runner.py:1108] Starting to load model ../../DeepSeek-OCR...
INFO 10-22 20:23:21 [config.py:3614] cudagraph sizes specified by model runner [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104] is overridden by config [96, 64, 32, 2, 4, 1, 104, 72, 40, 8, 80, 48, 16, 88, 24, 56]
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00
[rank0]: outputs_list = llm.generate(
[rank0]: ^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/utils.py", line 1196, in inner
[rank0]: return fn(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/entrypoints/llm.py", line 473, in generate
[rank0]: outputs = self._run_engine(use_tqdm=use_tqdm)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/entrypoints/llm.py", line 1423, in _run_engine
[rank0]: step_outputs = self.llm_engine.step()
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/engine/llm_engine.py", line 1412, in step
[rank0]: outputs = self.model_executor.execute_model(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 140, in execute_model
[rank0]: output = self.collective_rpc("execute_model",
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
[rank0]: answer = run_method(self.driver_worker, method, args, kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/utils.py", line 2456, in run_method
[rank0]: return func(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/worker/worker_base.py", line 420, in execute_model
[rank0]: output = self.model_runner.execute_model(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/worker/model_runner.py", line 1764, in execute_model
[rank0]: hidden_or_intermediate_states = model_executable(
[rank0]: ^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/Documents/development/DeepSeek-OCR/DeepSeek-OCR-master/DeepSeek-OCR-vllm/deepseek_ocr.py", line 543, in forward
[rank0]: vision_embeddings = self.get_multimodal_embeddings(**kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/Documents/development/DeepSeek-OCR/DeepSeek-OCR-master/DeepSeek-OCR-vllm/deepseek_ocr.py", line 503, in get_multimodal_embeddings
[rank0]: vision_embeddings = self._process_image_input(image_input)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/Documents/development/DeepSeek-OCR/DeepSeek-OCR-master/DeepSeek-OCR-vllm/deepseek_ocr.py", line 486, in _process_image_input
[rank0]: vision_features = self._pixel_values_to_embedding(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/Documents/development/DeepSeek-OCR/DeepSeek-OCR-master/DeepSeek-OCR-vllm/deepseek_ocr.py", line 394, in _pixel_values_to_embedding
[rank0]: local_features_1 = self.sam_model(patches)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/Documents/development/DeepSeek-OCR/DeepSeek-OCR-master/DeepSeek-OCR-vllm/deepencoder/sam_vary_sdpa.py", line 176, in forward
[rank0]: x = blk(x)
[rank0]: ^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/Documents/development/DeepSeek-OCR/DeepSeek-OCR-master/DeepSeek-OCR-vllm/deepencoder/sam_vary_sdpa.py", line 241, in forward
[rank0]: x = self.attn(x)
[rank0]: ^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/Documents/development/DeepSeek-OCR/DeepSeek-OCR-master/DeepSeek-OCR-vllm/deepencoder/sam_vary_sdpa.py", line 309, in forward
[rank0]: attn_bias = (rel_h + rel_w).view(B, self.num_heads, rel_h.size(2), rel_h.size(3) * rel_w.size(4))
[rank0]: ~~~~~~^~~~~~~
[rank0]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 352.00 MiB. GPU 0 has a total capacity of 11.75 GiB of which 278.19 MiB is free. Process 6151 has 102.00 MiB memory in use. Process 15532 has 21.52 MiB memory in use. Process 5363 has 23.02 MiB memory in use. Process 26235 has 26.90 MiB memory in use. Including non-PyTorch memory, this process has 10.65 GiB memory in use. Of the allocated memory 10.09 GiB is allocated by PyTorch, with 24.00 MiB allocated in private pools (e.g., CUDA Graphs), and 368.49 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Processed prompts: 0%| | 0/5 [00:00
[rank0]: llm = LLM(
[rank0]: ^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/utils.py", line 1161, in inner
[rank0]: return fn(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/entrypoints/llm.py", line 247, in __init__
[rank0]: self.llm_engine = LLMEngine.from_engine_args(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/engine/llm_engine.py", line 510, in from_engine_args
[rank0]: return engine_cls.from_vllm_config(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/engine/llm_engine.py", line 486, in from_vllm_config
[rank0]: return cls(
[rank0]: ^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/engine/llm_engine.py", line 278, in __init__
[rank0]: self._initialize_kv_caches()
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/engine/llm_engine.py", line 435, in _initialize_kv_caches
[rank0]: self.model_executor.initialize_cache(num_gpu_blocks, num_cpu_blocks)
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 123, in initialize_cache
[rank0]: self.collective_rpc("initialize_cache",
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 331, in collective_rpc
[rank0]: return self._run_workers(method, *args, **(kwargs or {}))
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/executor/mp_distributed_executor.py", line 185, in _run_workers
[rank0]: driver_worker_output = run_method(self.driver_worker, sent_method,
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/utils.py", line 2456, in run_method
[rank0]: return func(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/worker/worker.py", line 328, in initialize_cache
[rank0]: self._warm_up_model()
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/worker/worker.py", line 358, in _warm_up_model
[rank0]: self.model_runner.capture_model(self.gpu_cache)
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/worker/model_runner.py", line 1579, in capture_model
[rank0]: graph_runner.capture(**capture_inputs)
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/worker/model_runner.py", line 1960, in capture
[rank0]: output_hidden_or_intermediate_states = self.model(
[rank0]: ^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/Documents/development/DeepSeek-OCR/DeepSeek-OCR-master/DeepSeek-OCR-vllm/deepseek_ocr.py", line 544, in forward
[rank0]: inputs_embeds = self.get_input_embeddings(input_ids,
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/Documents/development/DeepSeek-OCR/DeepSeek-OCR-master/DeepSeek-OCR-vllm/deepseek_ocr.py", line 516, in get_input_embeddings
[rank0]: inputs_embeds = self.language_model.get_input_embeddings(input_ids)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/model_executor/models/deepseek.py", line 459, in get_input_embeddings
[rank0]: return self.model.get_input_embeddings(input_ids)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/model_executor/models/deepseek.py", line 360, in get_input_embeddings
[rank0]: return self.embed_tokens(input_ids)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/model_executor/layers/vocab_parallel_embedding.py", line 422, in forward
[rank0]: output = tensor_model_parallel_all_reduce(output_parallel)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/distributed/communication_op.py", line 13, in tensor_model_parallel_all_reduce
[rank0]: return get_tp_group().all_reduce(input_)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 353, in all_reduce
[rank0]: return torch.ops.vllm.all_reduce(input_,
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/torch/_ops.py", line 1123, in __call__
[rank0]: return self._op(*args, **(kwargs or {}))
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 109, in all_reduce
[rank0]: return group._all_reduce_out_place(tensor)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 359, in _all_reduce_out_place
[rank0]: return self.device_communicator.all_reduce(input_)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 63, in all_reduce
[rank0]: out = pynccl_comm.all_reduce(input_)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/distributed/device_communicators/pynccl.py", line 126, in all_reduce
[rank0]: self.nccl.ncclAllReduce(buffer_type(in_tensor.data_ptr()),
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 290, in ncclAllReduce
[rank0]: self.NCCL_CHECK(self._funcs["ncclAllReduce"](sendbuff, recvbuff, count,
[rank0]: File "/home/mahdi/.pyenv/versions/deepseek-ocr/lib/python3.12/site-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 256, in NCCL_CHECK
[rank0]: raise RuntimeError(f"NCCL error: {error_str}")
[rank0]: RuntimeError: NCCL error: invalid usage (run with NCCL_DEBUG=WARN for details)
[rank0]:[W1022 20:29:20.703079686 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
/home/mahdi/.pyenv/versions/3.12.9/lib/python3.12/multiprocessing/resource_tracker.py:255: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '

```