zilliztech / zilliztech/VectorDBBench

Cannot run vectordb-bench in offline env

Open
#620 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
438
Avg merge
1d 4h
Merged PRs (30d)
1

Description

Meet some issues when run vectordb-bench in offline env.

Reproduce steps:

  1. download cohere_medium_1m dataset
  2. run milvus standalone
  3. install vectordb-bench official pip package
pip install vectordb-bench
pip install vectordb-bench milvus
  1. run "init_bench", and then run task, flow will error out with this traceback:
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/caiyd/miniconda3/envs/vdb/lib/python3.11/site-packages/vectordb_bench/interface.py", line 182, in _async_task_v2
    case_res.metrics = runner.run(drop_old)
                       ^^^^^^^^^^^^^^^^^^^^
  File "/home/caiyd/miniconda3/envs/vdb/lib/python3.11/site-packages/vectordb_bench/backend/task_runner.py", line 120, in run
    self._pre_run(drop_old)
  File "/home/caiyd/miniconda3/envs/vdb/lib/python3.11/site-packages/vectordb_bench/backend/task_runner.py", line 112, in _pre_run
    self.ca.dataset.prepare(self.dataset_source, filters=self.ca.filters)
  File "/home/caiyd/miniconda3/envs/vdb/lib/python3.11/site-packages/vectordb_bench/backend/dataset.py", line 368, in prepare
    source.reader().read(
  File "/home/caiyd/miniconda3/envs/vdb/lib/python3.11/site-packages/vectordb_bench/backend/data_source.py", line 133, in read
    if (not local_file.exists()) or (not self.validate_file(remote_file, local_file)):
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/caiyd/miniconda3/envs/vdb/lib/python3.11/site-packages/vectordb_bench/backend/data_source.py", line 149, in validate_file
    info = self.fs.info(remote)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/caiyd/miniconda3/envs/vdb/lib/python3.11/site-packages/fsspec/asyn.py", line 118, in wrapper
    return sync(self.loop, func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  1. want to comment out this line "if (not local_file.exists()) or (not self.validate_file(remote_file, local_file)):",
    so I uninstall vectordb-bench, and install it locally, but it cannot start up
(vdb)  ~/work/zilliz/VectorDBBench/ [tags/v1.0.10] pip uninstall -y vectordb-bench               
Found existing installation: vectordb-bench 1.0.11
Uninstalling vectordb-bench-1.0.11:
  Successfully uninstalled vectordb-bench-1.0.11
(vdb)  ~/work/zilliz/VectorDBBench/ [tags/v1.0.10] 
(vdb)  ~/work/zilliz/VectorDBBench/ [tags/v1.0.10] pip cache purge                
Files removed: 286 (16.3 MB)
(vdb)  ~/work/zilliz/VectorDBBench/ [tags/v1.0.10] 
(vdb)  ~/work/zilliz/VectorDBBench/ [tags/v1.0.10] pip install -e ".[milvus]"
Obtaining file:///home/caiyd/work/zilliz/VectorDBBench
  Installing build dependencies ... done
......
55130281d18c9ffcdf77cae9
  Stored in directory: /tmp/pip-ephem-wheel-cache-n7c1wn_2/wheels/ac/03/a2/762233e655726a69db9d9b05989bd48a40b5ae020618dc4590
Successfully built vectordb-bench
Installing collected packages: vectordb-bench
Successfully installed vectordb-bench-1.0.10
(vdb)  ~/work/zilliz/VectorDBBench/ [tags/v1.0.10] 
(vdb)  ~/work/zilliz/VectorDBBench/ [tags/v1.0.10] pip list | grep vectordb-bench
130:vectordb-bench              1.0.10      /home/caiyd/work/zilliz/VectorDBBench
(vdb)  ~/work/zilliz/VectorDBBench/ [tags/v1.0.10] 
(vdb)  ~/work/zilliz/VectorDBBench/ [tags/v1.0.10] init_bench
Traceback (most recent call last):
  File "/home/caiyd/miniconda3/envs/vdb/bin/init_bench", line 3, in <module>
    from vectordb_bench.__main__ import main
  File "/home/caiyd/work/zilliz/VectorDBBench/vectordb_bench/__main__.py", line 6, in <module>
    from . import config
ImportError: cannot import name 'config' from 'vectordb_bench' (unknown location)
(vdb)  ~/work/zilliz/VectorDBBench/ [tags/v1.0.10] 

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with vectordb_bench/backend/data_source.py, especially read() and validate_file(), and reproduce the offline dataset flow through init_bench. Also inspect vectordb_bench/main.py and the package's config import for the editable-install failure. Done should mean the benchmark can use the downloaded dataset offline and init_bench starts from a local installation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.