zilliztech / zilliztech/VectorDBBench

s3 source files are not downloading in aws east region

Open
#313 0 comments 0 reactions 2 assignees View on GitHub

@XuanYang-cn is already working on this.

Since Apr 28, 2024.

Dominant language
Python
Stars
1.2k
Forks
438
Avg merge
1d 4h
Merged PRs (30d)
1

Description

HI Team,

We have setup the Vector DB Bench in aws east region. While running the test, it's unable to download the files from zilliz s3 location.

But same code is working in aws west2 region. Can you plz fix the issue.

2024-04-26 20:34:43,346 | INFO: Milvus create collection: VectorDBBenchCollection (milvus.py:55) (404)
2024-04-26 20:35:12,236 | WARNING: pre run case error: Connect timeout on endpoint URL: "https://s3.us-west-2.amazonaws.com/assets.zilliz.com/benchmark/cohere_medium_1m/test.parquet" (task_runner.py:92) (404)
2024-04-26 20:35:12,236 | WARNING: [1/1] case {'label': <CaseLabel.Performance: 2>, 'dataset': {'data': {'name': 'Cohere', 'size': 1000000, 'dim': 768, 'metric_type': <MetricType.COSINE: 'COSINE'>}}, 'db': 'Milvus-1M_HNSW_DEFAULT'} failed to run, reason=Connect timeout on endpoint URL: "https://s3.us-west-2.amazonaws.com/assets.zilliz.com/benchmark/cohere_medium_1m/test.parquet" (interface.py:187) (404)
Traceback (most recent call last):
File "/app/VectorDBBench/lib64/python3.11/site-packages/vectordb_bench/interface.py", line 168, in _async_task_v2
case_res.metrics = runner.run(drop_old)
^^^^^^^^^^^^^^^^^^^^
File "/app/VectorDBBench/lib64/python3.11/site-packages/vectordb_bench/backend/task_runner.py", line 96, in run
self._pre_run(drop_old)
File "/app/VectorDBBench/lib64/python3.11/site-packages/vectordb_bench/backend/task_runner.py", line 93, in _pre_run
raise e from None
File "/app/VectorDBBench/lib64/python3.11/site-packages/vectordb_bench/backend/task_runner.py", line 87, in _pre_run
self.ca.dataset.prepare(self.dataset_source)
File "/app/VectorDBBench/lib64/python3.11/site-packages/vectordb_bench/backend/dataset.py", line 202, in prepare
source.reader().read(
File "/app/VectorDBBench/lib64/python3.11/site-packages/vectordb_bench/backend/data_source.py", line 135, in read
if (not local_file.exists()) or (not self.validate_file(remote_file, local_file, check_etag)):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/VectorDBBench/lib64/python3.11/site-packages/vectordb_bench/backend/data_source.py", line 152, in validate_file
info = self.fs.info(remote)
^^^^^^^^^^^^^^^^^^^^
File "/app/VectorDBBench/lib64/python3.11/site-packages/fsspec/asyn.py", line 118, in wrapper
return sync(self.loop, func, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Thanks

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.