zilliztech / zilliztech/VectorDBBench

Few questions on Milvus when running LAION 100M large dataset

Open
#367 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
438
Avg merge
1d 4h
Merged PRs (30d)
1

Description

I am using VectorDBbench to perform and analyse the milvus capabilities before it handle our load at scale. ?
We are using 1 server with 4 NVIDIA L40S gpus and i have assigned 2 for querynode and 2 for indexnode.

When i ran the no filter search performance test index LAION 100M dataset and index type is DISKANN with K=100 and the entire setup just hung for like hours together in optmize state and wondering there is no more logs to see whats happening in this state ?

Few questions :

  1. What does optimise state really does in this case. ?
  2. Enlighten me here, i am thinking GPUs are no role to play in the optimize state.
  3. How long could we expect to this test to complete any rough ideas. ?
  4. Does the query and index gpus only play role , when there is any indexing happening and query happenings and how to check the GPU usage ? (I tried the nvidimia-smi) and there is no usage i observed.

Last status

#####. Data set #####
2024-08-30 13:05:46,369 | INFO: [1/1] start case: {'label': <CaseLabel.Performance: 2>, 'dataset': {'data': {'name': 'LAION', 'size': 100000000, 'dim': 768, 'metric_type': <MetricType.L2: 'L2'>}}, 'db': 'Milvus-r1u1'}, drop_old=True (interface.py:164) (2145320)


##### Current state #####
2024-08-31 05:00:37,072 | INFO: (SpawnProcess-1:1) Finish loading all dataset into VectorDB, dur=48396.15328549099 (serial_runner.py:61) (2584643)
2024-08-31 05:00:38,764 | INFO: Milvus post insert before optimize (milvus.py:101) (982732)

my helm values.yaml file looks like this :

`
indexNode:
resources:
requests:
nvidia.com/gpu: "2"
limits:
nvidia.com/gpu: "2"
queryNode:
resources:
requests:
nvidia.com/gpu: "2"
limits:
nvidia.com/gpu: "2"
mmap:
# Set memory mapping property for whole cluster
mmapEnabled: true
# Set memory-mapped directory path, if you leave mmapDirPath unspecified, the memory-mapped files will be stored in {localStorage.path}/ mmap by default.
mmapDirPath: /mnt/vector/clustersetup_files/

minio:
enabled: false

externalS3:
enabled: true
host: "xx..xx.xxx.xx"
port: "xx"
accessKey: "mykey"
secretKey: "myskey"
useSSL: false
bucketName: "milvusdb"
rootPath: ""
useIAM: false
cloudProvider: "aws"
iamEndpoint: ""`

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the logged post-insert optimize transition in milvus.py and the run completion in serial_runner.py, then compare them with the supplied Helm GPU and externalS3 settings. Done means reproducing or measuring the LAION 100M DISKANN run and documenting what optimize does, expected duration, and how GPU use can be checked.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.