zilliztech / zilliztech/VectorDBBench

Get an error :list index out of range, when running Milvus performance test using deep1b dataset.

Open
#688 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
438
Avg merge
1d 4h
Merged PRs (30d)
1

Description

Hi, there,

  I've just got an error when I run Milvus performance test with VectorDBBench using deep1b dataset.
  The error log shows that maybe somewhere occur index-out-of-range error.
  My Milvus's version is 2.5.21, and VectorDBBench's version is 1.0.16
  dataset files was downloaded from here: https://huggingface.co/datasets/open-vdb/deep-image-96-angular/tree/main
  Has someone met the same problem ? what should I do to solve it ?

  Here is my test configurations:
Image
and here is my task configurations:
Image Image

the error log shows below:

2025-12-26 19:24:03,174 | WARNING: search error: list index out of range, list index out of range (task_runner.py:269) (236456)
2025-12-26 19:24:03,174 | WARNING: Failed to run performance case, reason = list index out of range (task_runner.py:222) (236456)
Traceback (most recent call last):
File "/data/anaconda3/envs/milvus/lib/python3.12/site-packages/vectordb_bench/backend/task_runner.py", line 218, in run_perf_case
search_results = self.serial_search()
^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/milvus/lib/python3.12/site-packages/vectordb_bench/backend/task
runner.py", line 271, in serial_search
raise e from e
File "/data/anaconda3/envs/milvus/lib/python3.12/site-packages/vectordb_bench/backend/task
runner.py", line 267, in serial_search
results, _ = self.serial_search_runner.run()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/milvus/lib/python3.12/site-packages/vectordb_bench/backend/utils.py", line 43, in inner
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/milvus/lib/python3.12/site-packages/vectordb_bench/backend/runner/serial
runner.py", line 309, in run
return self.run_in_subprocess()
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/milvus/lib/python3.12/site-packages/vectordb_bench/backend/runner/serial
runner.py", line 300, in run_in_subprocess
return future.result()
^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/milvus/lib/python3.12/concurrent/futures/
base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/milvus/lib/python3.12/concurrent/futures/
base.py", line 401, in __get_result
raise self._exception
IndexError: list index out of range
2025-12-26 19:24:03,175 | WARNING: [1/1] case {'label': <CaseLabel.Performance: 2>, 'name': 'deep (Performace Case)', 'dataset': {'data': {'name': 'deep', 'size': 9990000, 'dim': 96, 'metric_type': <MetricType.COSINE: 'COSINE'>}}, 'db': 'Milvus'} failed to run, reason=list index out of range (interface.py:203) (236456)
Traceback (most recent call last):
File "/data/anaconda3/envs/milvus/lib/python3.12/site-packages/vectordb_bench/interface.py", line 182, in async_task_v2
case_res.metrics = runner.run(drop_old)
^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/milvus/lib/python3.12/site-packages/vectordb_bench/backend/task
runner.py", line 150, in run
return self.run_perf_case(drop_old)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/milvus/lib/python3.12/site-packages/vectordb_bench/backend/task
runner.py", line 224, in run_perf_case
raise e from None
File "/data/anaconda3/envs/milvus/lib/python3.12/site-packages/vectordb_bench/backend/task
runner.py", line 218, in run_perf_case
search_results = self.serial_search()
^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/milvus/lib/python3.12/site-packages/vectordb_bench/backend/task
runner.py", line 271, in serial_search
raise e from e
File "/data/anaconda3/envs/milvus/lib/python3.12/site-packages/vectordb_bench/backend/task
runner.py", line 267, in serial_search
results, _ = self.serial_search_runner.run()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/milvus/lib/python3.12/site-packages/vectordb_bench/backend/utils.py", line 43, in inner
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/milvus/lib/python3.12/site-packages/vectordb_bench/backend/runner/serial
runner.py", line 309, in run
return self.run_in_subprocess()
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/milvus/lib/python3.12/site-packages/vectordb_bench/backend/runner/serial
runner.py", line 300, in run_in_subprocess
return future.result()
^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/milvus/lib/python3.12/concurrent/futures/
base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/milvus/lib/python3.12/concurrent/futures/
base.py", line 401, in __get_result
raise self._exception
IndexError: list index out of range

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the deep dataset performance case, then inspect vectordb_bench/backend/task_runner.py around lines 267-271 and backend/runner/serial_runner.py around lines 300-309 from the traceback. Trace the source of the IndexError and make the case complete without failing; verify by rerunning the same Milvus configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases, performance, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.