mlcommons / mlcommons/storage

Running KVcache after VectorDB causes a hang in cluster_collector.py

Open
#843 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
203
Forks
67
Avg merge
20m
Merged PRs (30d)
8

Description

I found in the course of testing that if I run the kvcache benchmark after installing and running the vectordb tests that a hang will occur at the beginning when the benchmark is setting up the collector. Note that this only occurs on multiple client runs, running with a single client doesn't show this behavior. I have a workaround but wanted to bring this up so it could get addressed in future versions or in the docs.

In this condition, kcking of kvcache with 2 or more hosts will get to the following line and hang, ultimately timing out and failing.

CLI: uv run mlpstorage open kvcache run --hosts wsclient01 wsclient02 --cache-dir /mnt/kv --results-dir /mnt/logs/0811

Collecting via MPI... 0:00:002026-08-11 13:51:58|INFO:cluster_collector:3366: Collector script staged on wsclient02:/mnt/logs/0811/open/Nutanix/results/wsclient01/kv_cache/tiny-1b/run/20260811_135054/collector-staging
2026-08-11 13:51:58|INFO:cluster_collector:3457: Running MPI collection across 2 host(s)

I ran with --debug and saw that the hang happened during this mpirun command:

mpirun -n 2 -host wsclient01:1,wsclient02:1 --bind-to none --map-by node /home/nutanix/mlperf3/storage/.venv/bin/python3 /mnt/logs/0811/open/Nutanix/results/wsclient01/kv_cache/tiny-1b/run/20260811_135054/collector-staging/mlps_collector.py /mnt/logs/0811/open/Nutanix/results/wsclient01/kv_cache/tiny-1b/run/20260811_135054/collector-staging/cluster_info.json

which hung and showed this warning message:


WARNING: Open MPI accepted a TCP connection from what appears to be a
another Open MPI process but cannot find a corresponding process
entry for that peer.

This attempted connection will be ignored; your MPI job may or may not
continue properly.

Local host: wsclient02
PID: 6181

Searching around this is likely caused by a multi-home issue where MPI is trying to use a particular network interface while the partner listening on a different one.

The only thing generally different about these clients is that I used them for running Vectordb testing before running kvcache so there is a docker0 interface and a docker bridge interface on each one.

After deleting the two interfaces on each client (docker0 and br-xxxxxxxx) the benchmark executed fine.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in cluster_collector.py around the reported collection and MPI launch points at lines 3366 and 3457, then inspect mlps_collector.py and reproduce the command with two hosts after running the VectorDB tests. Compare behavior with the docker0 and bridge interfaces present versus removed. Done means the multi-client KVcache benchmark completes reliably, or the required interface workaround is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.