zilliztech / zilliztech/VectorDBBench

Title: Milvus GPU (CAGRA) benchmark stuck at “Milvus optimizing before search” Description

Open
#615 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
438
Avg merge
1d 4h
Merged PRs (30d)
1

Description

I’m benchmarking Milvus GPU (CAGRA index) using VectorDBBench, and the benchmark seems to hang indefinitely during the “optimizing before search” phase.

Issue: Benchmark stuck at “Milvus optimizing before search”
Milvus log :

...
failed to create index, C Runtime Exception:  => failed to build index, raft inner error at ...VectorMemIndex.cpp:276
segcore error[segcoreCode=2004]

The process freezes after showing this log message:

2025-10-16 07:08:58,237 | INFO: Milvus optimizing before search (milvus.py:145) (2752617)

It has been running for more than 2 hours without progressing to the search stage.

Observations:

GPU utilization stays low (nvidia-smi shows near-idle usage)

No new logs from Milvus container after this message

The benchmark does not terminate or error out

Steps to reproduce:

Start Milvus GPU standalone:

wget https://github.com/milvus-io/milvus/releases/download/v2.6.3/milvus-standalone-docker-compose-gpu.yml -O docker-compose.yml
docker compose up -d

Start the benchmark from VectorDBBench UI.

Observe that it stops at the “optimizing before search” phase and never completes.

Expected:
Benchmark should move from optimization to search and finish for all dataset sizes.

Actual:
It remains stuck in optimization indefinitely.

Environment
OS: Ubuntu 22.04
Python: 3.12 (venv)
VectorDBBench: 1.0.10
Milvus: 2.6.3
GPU: NVIDIA RTX (please specify your model)
Milvus Port: 19530
Datasets Tested: 100k, 250k, 1M, 5M

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read the benchmark path at milvus.py:145 and the reported Milvus gpu_raft.h and VectorMemIndex.cpp errors; reproduce with docker-compose.yml and compare behavior across dataset sizes. Done means the benchmark advances through search or exits with a clear error instead of remaining indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker-compose, python
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.