[Feature] Cannot find a way to disable custom_all_reduce: getting "[TM][WARNING] Device 0 peer access Device 1 is not available." messages

### Motivation

This is happening when I try to run any AWQ quant of InternVL.
I have a dual GPU setup.

It used to work with an older install/environment, but now I updated to the latest lmdeploy version, and I am getting these warnings and Out of Memory errors, likely due to the problem reported in the warnings. Searching online, I could find that disabling use_custom_all_reduce on TensorRT-LLM solves the problem, which likely means that lmdeply has custom_all_reduce On by default, but I need this off.

Can anyone tell me how to disable CustomAllReduce from lmdeploy?

Looking at the source code, I could find two references to that:

https://github.com/InternLM/lmdeploy/blob/0d600d2cb0e70ad85b56ea71b5ee5e8b77db8c65/src/turbomind/triton_backend/llama/LlamaTritonModel.cc#L454
https://github.com/InternLM/lmdeploy/blob/0d600d2cb0e70ad85b56ea71b5ee5e8b77db8c65/src/turbomind/utils/custom_ar_comm.cc#L125

Logs:

```

[TM][WARNING] [LlamaTritonModel] max_context_token_num = 32776.
2024-07-18 08:30:29,003 - lmdeploy - WARNING - get 483 model params
2024-07-18 08:30:55,337 - lmdeploy - INFO - updated backend_config=TurbomindEngineConfig(model_name=None, model_format='awq', tp=1, session_len=None, max_batch_size=128, cache_max_entry_count=0.8, cache_block_seq_len=64, enable_prefix_caching=False, quant_policy=0, rope_scaling_factor=0.0, use_logn_attn=False, download_dir=None, revision=None, max_prefill_token_num=8192, num_tokens_per_iter=0, max_prefill_iters=1)
[TM][WARNING] Device 0 peer access Device 1 is not available.
[WARNING] gemm_config.in is not found; using default GEMM algo
[TM][INFO] NCCL group_id = 0
[TM][INFO] [BlockManager] block_size = 12 MB
[TM][INFO] [BlockManager] max_block_count = 71
[TM][INFO] [BlockManager] chunk_size = 71
[TM][WARNING] No enough blocks for session_len (32776), session_len truncated to 4544.
Exception in thread Thread-4 (_create_model_instance):
Traceback (most recent call last):
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run()
File "/usr/lib/python3.12/threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "/run/media/computer/5dcc41df-7194-4e57-a28f-833dc5ce81bb/captioner/venv/lib/python3.12/site-packages/lmdeploy/turbomind/turbomind.py", line 398, in _create_model_instance
model_inst = self.tm_model.model_comm.create_model_instance(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: [TM][ERROR] CUDA runtime error: out of memory /lmdeploy/src/turbomind/utils/allocator.h:231

[TM][WARNING] Device 0 peer access Device 1 is not available.
[WARNING] gemm_config.in is not found; using default GEMM algo
[TM][INFO] NCCL group_id = 0
[TM][INFO] [BlockManager] block_size = 12 MB
[TM][INFO] [BlockManager] max_block_count = 0
[TM][INFO] [BlockManager] chunk_size = 0
[TM][WARNING] No enough blocks for session_len (32776), session_len truncated to 0.
[TM][INFO] LlamaBatch::Start()
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
[TM][WARNING] Device 0 peer access Device 1 is not available.
2024-07-18 08:30:55,596 - lmdeploy - INFO - start ImageEncoder._forward_loop
2024-07-18 08:30:55,596 - lmdeploy - INFO - ImageEncoder received 1 images, left 1 images.
2024-07-18 08:30:55,596 - lmdeploy - INFO - ImageEncoder process 1 images, left 0 images.
Exception in callback _raise_exception_on_finish() at /run/media/computer/5dcc41df-7194-4e57-a28f-833dc5ce81bb/captioner/venv/lib/python3.12/site-packages/lmdeploy/vl/engine.py:19
handle: ) at /run/media/computer/5dcc41df-7194-4e57-a28f-833dc5ce81bb/captioner/venv/lib/python3.12/site-packages/lmdeploy/vl/engine.py:19>
Traceback (most recent call last):
File "/usr/lib/python3.12/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "/run/media/computer/5dcc41df-7194-4e57-a28f-833dc5ce81bb/captioner/venv/lib/python3.12/site-packages/lmdeploy/vl/engine.py", line 26, in _raise_exception_on_finish
raise e
File "/run/media/computer/5dcc41df-7194-4e57-a28f-833dc5ce81bb/captioner/venv/lib/python3.12/site-packages/lmdeploy/vl/engine.py", line 22, in _raise_exception_on_finish
task.result()
File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/run/media/computer/5dcc41df-7194-4e57-a28f-833dc5ce81bb/captioner/venv/lib/python3.12/site-packages/lmdeploy/vl/engine.py", line 151, in forward
outputs = self.model.forward(inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/run/media/computer/5dcc41df-7194-4e57-a28f-833dc5ce81bb/captioner/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/run/media/computer/5dcc41df-7194-4e57-a28f-833dc5ce81bb/captioner/venv/lib/python3.12/site-packages/lmdeploy/vl/model/internvl.py", line 172, in forward
return self._forward_func(images)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/run/media/computer/5dcc41df-7194-4e57-a28f-833dc5ce81bb/captioner/venv/lib/python3.12/site-packages/lmdeploy/vl/model/internvl.py", line 153, in _forward_v1_5
outputs = self.model.extract_feature(outputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/computer/.cache/huggingface/modules/transformers_modules/InternVL2-26B-AWQ/modeling_internvl_chat.py", line 180, in extract_feature
vit_embeds = self.vision_model(
^^^^^^^^^^^^^^^^^^
File "/run/media/computer/5dcc41df-7194-4e57-a28f-833dc5ce81bb/captioner/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/run/media/computer/5dcc41df-7194-4e57-a28f-833dc5ce81bb/captioner/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/computer/.cache/huggingface/modules/transformers_modules/InternVL2-26B-AWQ/modeling_intern_vit.py", line 415, in forward
hidden_states = self.embeddings(pixel_values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/run/media/computer/5dcc41df-7194-4e57-a28f-833dc5ce81bb/captioner/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/run/media/computer/5dcc41df-7194-4e57-a28f-833dc5ce81bb/captioner/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/computer/.cache/huggingface/modules/transformers_modules/InternVL2-26B-AWQ/modeling_intern_vit.py", line 169, in forward
patch_embeds = self.patch_embedding(pixel_values) # shape = [*, channel, width, height]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/run/media/computer/5dcc41df-7194-4e57-a28f-833dc5ce81bb/captioner/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/run/media/computer/5dcc41df-7194-4e57-a28f-833dc5ce81bb/captioner/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/run/media/computer/5dcc41df-7194-4e57-a28f-833dc5ce81bb/captioner/venv/lib/python3.12/site-packages/torch/nn/modules/conv.py", line 460, in forward
return self._conv_forward(input, self.weight, self.bias)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/run/media/computer/5dcc41df-7194-4e57-a28f-833dc5ce81bb/captioner/venv/lib/python3.12/site-packages/torch/nn/modules/conv.py", line 456, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

```

### Related resources

Here is my GPU topology:

```

(venv) [computer@computer captioner]$ nvidia-smi topo -m
GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X PHB 0-15 0 N/A
GPU1 PHB X 0-15 0 N/A

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

```