cannot run internode test on 2 Node setup , with each having Two GPU - num_ranks > NUM_MAX_NVL_PEERS or low_latency_mode'

I have below setup and what to run test_internode.py

but getting below error

**Assertion error /home/host/alok/deepep/DeepEP/csrc/deep_ep.cpp:48 'num_ranks > NUM_MAX_NVL_PEERS or low_latency_mode'**

NCCL_DEBUG=INFO MASTER_ADDR=100.100.100.1 WORLD_SIZE=2 RANK=0 python test_internode.py --num-processes=2
NCCL_DEBUG=INFO MASTER_ADDR=100.100.100.1 WORLD_SIZE=2 RANK=1 python test_internode.py --num-processes=2

```
(venv) host@host-AS-5126GS-TNRT:~/alok/deepep/DeepEP$ NCCL_DEBUG=INFO MASTER_ADDR=100.100.100.1 WORLD_SIZE=2 RANK=0 python test_internode.py --num-processes=2
host-AS-5126GS-TNRT:160539:160539 [0] NCCL INFO Bootstrap: Using enp81s0f0np0:10.28.38.82<0>
host-AS-5126GS-TNRT:160539:160539 [0] NCCL INFO cudaDriverVersion 13000
host-AS-5126GS-TNRT:160539:160539 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
host-AS-5126GS-TNRT:160540:160540 [1] NCCL INFO cudaDriverVersion 13000
host-AS-5126GS-TNRT:160540:160540 [1] NCCL INFO Bootstrap: Using enp81s0f0np0:10.28.38.82<0>
host-AS-5126GS-TNRT:160540:160540 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
host-AS-5126GS-TNRT:160540:160540 [1] NCCL INFO Comm config Blocking set to 1
host-AS-5126GS-TNRT:160539:160539 [0] NCCL INFO Comm config Blocking set to 1
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO NET/Plugin: Could not find: libnccl-net.so.
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO NET/IB : Using [0]rocep113s0:1/RoCE [1]mlx5_0:1/RoCE [RO]; OOB enp81s0f0np0:10.28.38.82<0>
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO Initialized NET plugin IB
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO Assigned NET plugin IB to comm
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO Using network IB
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO DMA-BUF is available on GPU device 1
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO ncclCommInitRankConfig comm 0x235bc8f0 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId f1000 commId 0xfb702ae9b623d36f - Init START
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so.
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO NET/IB : Using [0]rocep113s0:1/RoCE [1]mlx5_0:1/RoCE [RO]; OOB enp81s0f0np0:10.28.38.82<0>
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Initialized NET plugin IB
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Assigned NET plugin IB to comm
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Using network IB
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO DMA-BUF is available on GPU device 0
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO ncclCommInitRankConfig comm 0x45ff2410 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId e1000 commId 0xfb702ae9b623d36f - Init START
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO Bootstrap timings total 0.670492 (create 0.000033, send 0.000136, recv 0.502401, ring 0.156162, delay 0.000000)
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Bootstrap timings total 0.535270 (create 0.000034, send 0.000133, recv 0.000200, ring 0.018268, delay 0.000000)
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Setting affinity for GPU 0 to 64-127,192-238,240-255
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO Setting affinity for GPU 1 to 64-127,192-238,240-255
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO comm 0x235bc8f0 rank 1 nRanks 4 nNodes 2 localRanks 2 localRank 1 MNNVL 0
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO P2P Chunksize set to 131072
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO comm 0x45ff2410 rank 0 nRanks 4 nNodes 2 localRanks 2 localRank 0 MNNVL 0
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Channel 00/02 : 0 1 2 3
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Channel 01/02 : 0 1 2 3
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Trees [0] 1/2/-1->0->-1 [1] 1/-1/-1->0->2
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO P2P Chunksize set to 131072
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
host-AS-5126GS-TNRT:160540:160796 [1] NCCL INFO [Proxy Service] Device 1 CPU core 207
host-AS-5126GS-TNRT:160540:160797 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 214
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
host-AS-5126GS-TNRT:160539:160799 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 97
host-AS-5126GS-TNRT:160539:160798 [0] NCCL INFO [Proxy Service] Device 0 CPU core 95
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO CC Off, workFifoBytes 1048576
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO ncclCommInitRankConfig comm 0x235bc8f0 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId f1000 commId 0xfb702ae9b623d36f - Init COMPLETE
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO ncclCommInitRankConfig comm 0x45ff2410 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId e1000 commId 0xfb702ae9b623d36f - Init COMPLETE
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 4 total 0.90 (kernels 0.12, alloc 0.10, bootstrap 0.67, allgathers 0.01, topo 0.01, graphs 0.00, connections 0.00, rest 0.00)
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 4 total 0.80 (kernels 0.15, alloc 0.10, bootstrap 0.54, allgathers 0.01, topo 0.01, graphs 0.00, connections 0.00, rest 0.00)
host-AS-5126GS-TNRT:160540:160540 [1] NCCL INFO Comm config Blocking set to 1
host-AS-5126GS-TNRT:160539:160539 [0] NCCL INFO Comm config Blocking set to 1
host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO Assigned NET plugin IB to comm
host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO Using network IB
host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO DMA-BUF is available on GPU device 1
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO Assigned NET plugin IB to comm
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO Using network IB
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO DMA-BUF is available on GPU device 0
host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO ncclCommSplit comm 0x238d0a60 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId f1000 parent 0x235bc8f0 splitCount 1 color 2003953581 key 1- Init START
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO ncclCommSplit comm 0x46853990 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId e1000 parent 0x45ff2410 splitCount 1 color 2003953581 key 0- Init START
host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO Setting affinity for GPU 1 to 64-127,192-238,240-255
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO Setting affinity for GPU 0 to 64-127,192-238,240-255
host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO comm 0x238d0a60 rank 1 nRanks 4 nNodes 2 localRanks 2 localRank 1 MNNVL 0
host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO P2P Chunksize set to 131072
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO comm 0x46853990 rank 0 nRanks 4 nNodes 2 localRanks 2 localRank 0 MNNVL 0
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO Channel 00/02 : 0 1 2 3
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO Channel 01/02 : 0 1 2 3
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO Trees [0] 1/2/-1->0->-1 [1] 1/-1/-1->0->2
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO P2P Chunksize set to 131072
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
host-AS-5126GS-TNRT:160540:160806 [1] NCCL INFO [Proxy Service] Device 1 CPU core 203
host-AS-5126GS-TNRT:160540:160807 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 81
host-AS-5126GS-TNRT:160539:160808 [0] NCCL INFO [Proxy Service] Device 0 CPU core 90
host-AS-5126GS-TNRT:160539:160809 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 97
host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO CC Off, workFifoBytes 1048576
host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO ncclCommSplit comm 0x238d0a60 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId f1000 parent 0x235bc8f0 splitCount 1 color 2003953581 key 1 - Init COMPLETE
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO ncclCommSplit comm 0x46853990 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId e1000 parent 0x45ff2410 splitCount 1 color 2003953581 key 0 - Init COMPLETE
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO Init timings - ncclCommSplit: rank 0 nranks 4 total 0.04 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.01, topo 0.01, graphs 0.00, connections 0.00, rest 0.02)
host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO Init timings - ncclCommSplit: rank 1 nranks 4 total 0.04 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.01, topo 0.01, graphs 0.00, connections 0.00, rest 0.02)
[rank0]:[W1103 14:04:39.231865515 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
host-AS-5126GS-TNRT:160540:160811 [1] NCCL INFO comm 0x238d0a60 rank 1 nranks 4 cudaDev 1 busId f1000 - Abort COMPLETE
host-AS-5126GS-TNRT:160539:160813 [0] NCCL INFO comm 0x46853990 rank 0 nranks 4 cudaDev 0 busId e1000 - Abort COMPLETE
host-AS-5126GS-TNRT:160540:160843 [1] NCCL INFO comm 0x235bc8f0 rank 1 nranks 4 cudaDev 1 busId f1000 - Abort COMPLETE
host-AS-5126GS-TNRT:160539:160845 [0] NCCL INFO comm 0x45ff2410 rank 0 nranks 4 cudaDev 0 busId e1000 - Abort COMPLETE
W1103 14:04:41.835000 160472 venv/lib/python3.12/site-packages/torch/multiprocessing/spawn.py:174] Terminating process 160539 via signal SIGTERM
Traceback (most recent call last):
File "/home/host/alok/deepep/DeepEP/test_internode.py", line 375, in
torch.multiprocessing.spawn(test_loop, args=(num_processes, args), nprocs=num_processes)
File "/home/host/alok/deepep/DeepEP/venv/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 364, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/host/alok/deepep/DeepEP/venv/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 320, in start_processes
while not context.join():
^^^^^^^^^^^^^^
File "/home/host/alok/deepep/DeepEP/venv/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 220, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 1 terminated with the following error:
Traceback (most recent call last):
File "/home/host/alok/deepep/DeepEP/venv/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 95, in _wrap
fn(i, *args)
File "/home/host/alok/deepep/DeepEP/test_internode.py", line 307, in test_loop
buffer = deep_ep.Buffer(group,
^^^^^^^^^^^^^^^^^^^^^
File "/home/host/alok/deepep/DeepEP/deep_ep/buffer.py", line 90, in __init__
self.runtime = deep_ep_cpp.Buffer(self.rank, self.group_size, num_nvl_bytes, num_rdma_bytes, low_latency_mode, explicitly_destroy,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Failed: Assertion error /home/host/alok/deepep/DeepEP/csrc/deep_ep.cpp:48 'num_ranks > NUM_MAX_NVL_PEERS or low_latency_mode'
```