Running KVcache after VectorDB causes a hang in cluster_collector.py
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 203
- Forks
- 67
- Avg merge
- 20m
- Merged PRs (30d)
- 8
Description
I found in the course of testing that if I run the kvcache benchmark after installing and running the vectordb tests that a hang will occur at the beginning when the benchmark is setting up the collector. Note that this only occurs on multiple client runs, running with a single client doesn't show this behavior. I have a workaround but wanted to bring this up so it could get addressed in future versions or in the docs.
In this condition, kcking of kvcache with 2 or more hosts will get to the following line and hang, ultimately timing out and failing.
CLI: uv run mlpstorage open kvcache run --hosts wsclient01 wsclient02 --cache-dir /mnt/kv --results-dir /mnt/logs/0811
Collecting via MPI... 0:00:002026-08-11 13:51:58|INFO:cluster_collector:3366: Collector script staged on wsclient02:/mnt/logs/0811/open/Nutanix/results/wsclient01/kv_cache/tiny-1b/run/20260811_135054/collector-staging
2026-08-11 13:51:58|INFO:cluster_collector:3457: Running MPI collection across 2 host(s)
I ran with --debug and saw that the hang happened during this mpirun command:
mpirun -n 2 -host wsclient01:1,wsclient02:1 --bind-to none --map-by node /home/nutanix/mlperf3/storage/.venv/bin/python3 /mnt/logs/0811/open/Nutanix/results/wsclient01/kv_cache/tiny-1b/run/20260811_135054/collector-staging/mlps_collector.py /mnt/logs/0811/open/Nutanix/results/wsclient01/kv_cache/tiny-1b/run/20260811_135054/collector-staging/cluster_info.json
which hung and showed this warning message:
WARNING: Open MPI accepted a TCP connection from what appears to be a
another Open MPI process but cannot find a corresponding process
entry for that peer.
This attempted connection will be ignored; your MPI job may or may not
continue properly.
Local host: wsclient02
PID: 6181
Searching around this is likely caused by a multi-home issue where MPI is trying to use a particular network interface while the partner listening on a different one.
The only thing generally different about these clients is that I used them for running Vectordb testing before running kvcache so there is a docker0 interface and a docker bridge interface on each one.
After deleting the two interfaces on each client (docker0 and br-xxxxxxxx) the benchmark executed fine.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in cluster_collector.py around the reported collection and MPI launch points at lines 3366 and 3457, then inspect mlps_collector.py and reproduce the command with two hosts after running the VectorDB tests. Compare behavior with the docker0 and bridge interfaces present versus removed. Done means the multi-client KVcache benchmark completes reliably, or the required interface workaround is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100