[KVCache] The AIBRIX_KV_CHE-OL_INFINISTORE_VISIBLE_deV_LIST setting is not taking effect

🚀 Feature Description and Motivation

@Jeffwan
My machine only have two card can use.First I set AIBRIX_KV_CACHE_OL_INFINISTORE_VISIBLE_DEV_LIST:mlx5_0,mlx5_2, it happend the next error. I change AIBRIX_KV_CACHE_OL_INFINISTORE_VISIBLE_DEV_LIST:mlx5_0:0,mlx5_2:0 , it happend the similiar error.Otherwise, I only have one A100 test machine.

[2025-06-03 06:31:02.209] [infini] [info] ServerConfig: service_port=12345, manage_port=8088, log_level='info', dev_name='mlx5_1', ib_port=1, link_type='Ethernet', prealloc_size=108, minimal_allocate_size=64, auto_increase=False, evict_min_threshold=0.6, evict_max_threshold=0.8, evict_interval=5, hint_gid_index=7
[2025-06-03 06:31:02.209] [infini] [info] open rdma device mlx5_1, link_type Ethernet, hint_gid_index 7,
[2025-06-03 06:31:02.210] [infini] [info] Interrupt signal (11) received.
[2025-06-03 06:31:02.210] [infini] [error] [utils.cpp:99] Stacktrace:
 0# signal_handler(int) in /usr/local/lib/python3.10/dist-packages/infinistore/_infinistore.cpython-310-x86_64-linux-gnu.so
 1# 0x00007FE78FFC3520 in /usr/lib/x86_64-linux-gnu/libc.so.6
 2# 0x00007FE79011E7FD in /usr/lib/x86_64-linux-gnu/libc.so.6
 3# decltype ({parm#1}(0)) fmt::v11::basic_format_arg<fmt::v11::context>::visit<fmt::v11::detail::default_arg_formatter<char> >(fmt::v11::detail::default_arg_formatter<char>&&) const in /usr/local/lib/python3.10/dist-packages/infinistore/_infinistore.cpython-310-x86_64-linux-gnu.so
 4# char const* fmt::v11::detail::parse_replacement_field<char, fmt::v11::detail::format_handler<char>&>(char const*, char const*, fmt::v11::detail::format_handler<char>&) in /usr/local/lib/python3.10/dist-packages/infinistore/_infinistore.cpython-310-x86_64-linux-gnu.so
 5# fmt::v11::detail::vformat_to(fmt::v11::detail::buffer<char>&, fmt::v11::basic_string_view<char>, fmt::v11::basic_format_args<fmt::v11::context>, fmt::v11::detail::locale_ref) in /usr/local/lib/python3.10/dist-packages/infinistore/_infinistore.cpython-310-x86_64-linux-gnu.so
 6# open_rdma_device(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, rdma_device*) in /usr/local/lib/python3.10/dist-packages/infinistore/_infinistore.cpython-310-x86_64-linux-gnu.so
 7# register_server(unsigned long, ServerConfig) in /usr/local/lib/python3.10/dist-packages/infinistore/_infinistore.cpython-310-x86_64-linux-gnu.so
 8# 0x00007FE78FAE67C0 in /usr/local/lib/python3.10/dist-packages/infinistore/_infinistore.cpython-310-x86_64-linux-gnu.so
 9# 0x00007FE78FADA584 in /usr/local/lib/python3.10/dist-packages/infinistore/_infinistore.cpython-310-x86_64-linux-gnu.so
10# 0x000055D6FF65EE12 in /usr/bin/python3
11# _PyObject_MakeTpCall in /usr/bin/python3
12# _PyEval_EvalFrameDefault in /usr/bin/python3
13# _PyFunction_Vectorcall in /usr/bin/python3
14# _PyEval_EvalFrameDefault in /usr/bin/python3
15# _PyFunction_Vectorcall in /usr/bin/python3
16# _PyEval_EvalFrameDefault in /usr/bin/python3
17# 0x000055D6FF72DF56 in /usr/bin/python3
18# PyEval_EvalCode in /usr/bin/python3
19# 0x000055D6FF754808 in /usr/bin/python3
20# 0x000055D6FF74F00F in /usr/bin/python3
21# 0x000055D6FF7545A5 in /usr/bin/python3
22# _PyRun_SimpleFileObject in /usr/bin/python3
23# _PyRun_AnyFileObject in /usr/bin/python3
24# Py_RunMain in /usr/bin/python3
25# Py_BytesMain in /usr/bin/python3
26# 0x00007FE78FFAAD90 in /usr/lib/x86_64-linux-gnu/libc.so.6
27# __libc_start_main in /usr/lib/x86_64-linux-gnu/libc.so.6
28# _start in /usr/bin/python3

Use Case

Env:

mlx5_0 port 1 ==> lan2 (Up)
mlx5_1 port 1 ==> lan3 (Down)
mlx5_2 port 1 ==> lan4 (Up)
mlx5_3 port 1 ==> lan5 (Down)

          env:
            - name: VLLM_USE_V1
              value: "0"
            - name: AIBRIX_KV_CACHE_OL_L1_CACHE_ENABLED
              value: "0"
            - name: AIBRIX_KV_CACHE_OL_L2_CACHE_BACKEND
              value: "infinistore"
            - name: AIBRIX_KV_CACHE_OL_INFINISTORE_CONNECTION_TYPE
              value: "RDMA"
            - name: AIBRIX_KV_CACHE_OL_INFINISTORE_IB_PORT
              value: "1"
            - name: AIBRIX_KV_CACHE_OL_INFINISTORE_LINK_TYPE
              value: "Ethernet"
            - name: AIBRIX_KV_CACHE_OL_INFINISTORE_VISIBLE_DEV_LIST
              value: "mlx5_0,mlx5_2"
            - name: AIBRIX_KV_CACHE_OL_META_SERVICE_BACKEND
              value: "redis"
            - name: AIBRIX_KV_CACHE_OL_META_SERVICE_URL
              value: "redis://kvcache-cluster-redis:6379"
            - name: AIBRIX_KV_CACHE_OL_META_SERVICE_CLUSTER_META_KEY
              value: "kvcache_nodes"
            - name: VLLM_RPC_TIMEOUT
              value: "1000000"

mlx5_0:
                        GID[  0]:               fe80:0000:0000:0000:8ae9:a4ff:... RoCE v1
                        GID[  1]:               fe80::8ae9:a4ff:fe53:49dc, RoCE v2
                        GID[  2]:               0000:0000:0000:0000:0000:ffff:..., RoCE v1
                        GID[  3]:               ::ffff:10.163.72.39, RoCE v2
mlx5_1:
                        GID[  0]:               fe80:0000:0000:0000:8ae9:a4ff:..., RoCE v1
                        GID[  1]:               fe80::8ae9:a4ff:fe53:49dd, RoCE v2
mlx5_2:
                    GID[  0]:               fe80:0000:0000:0000:0ac0:ebff:..., RoCE v1
                        GID[  1]:               fe80::ac0:ebff:fe94:e27a, RoCE v2
mlx5_3:
                        GID[  0]:               fe80:0000:0000:0000:0ac0:...., RoCE v1
                        GID[  1]:               fe80::ac0:ebff:fe94:e27b, RoCE v2

#kv cache annotations
annotations:
    kvcache.orchestration.aibrix.ai/backend: infinistore
    infinistore.kvcache.orchestration.aibrix.ai/link-type: "Ethernet"
    infinistore.kvcache.orchestration.aibrix.ai/hint-gid-index: "7"

Proposed Solution

No response

No source file or test is named. Start by reproducing the RDMA crash with AIBRIX_KV_CACHE_OL_INFINISTORE_VISIBLE_DEV_LIST set to mlx5_0,mlx5_2 and trace how the setting reaches the Infinistore device-opening path; done means the selected devices are used without opening mlx5_1 or crashing.