zilliztech / zilliztech/VectorDBBench
Few questions on Milvus when running LAION 100M large dataset
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 438
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 1
Description
I am using VectorDBbench to perform and analyse the milvus capabilities before it handle our load at scale. ?
We are using 1 server with 4 NVIDIA L40S gpus and i have assigned 2 for querynode and 2 for indexnode.
When i ran the no filter search performance test index LAION 100M dataset and index type is DISKANN with K=100 and the entire setup just hung for like hours together in optmize state and wondering there is no more logs to see whats happening in this state ?
Few questions :
- What does optimise state really does in this case. ?
- Enlighten me here, i am thinking GPUs are no role to play in the optimize state.
- How long could we expect to this test to complete any rough ideas. ?
- Does the query and index gpus only play role , when there is any indexing happening and query happenings and how to check the GPU usage ? (I tried the nvidimia-smi) and there is no usage i observed.
Last status
#####. Data set #####
2024-08-30 13:05:46,369 | INFO: [1/1] start case: {'label': <CaseLabel.Performance: 2>, 'dataset': {'data': {'name': 'LAION', 'size': 100000000, 'dim': 768, 'metric_type': <MetricType.L2: 'L2'>}}, 'db': 'Milvus-r1u1'}, drop_old=True (interface.py:164) (2145320)
##### Current state #####
2024-08-31 05:00:37,072 | INFO: (SpawnProcess-1:1) Finish loading all dataset into VectorDB, dur=48396.15328549099 (serial_runner.py:61) (2584643)
2024-08-31 05:00:38,764 | INFO: Milvus post insert before optimize (milvus.py:101) (982732)
my helm values.yaml file looks like this :
`
indexNode:
resources:
requests:
nvidia.com/gpu: "2"
limits:
nvidia.com/gpu: "2"
queryNode:
resources:
requests:
nvidia.com/gpu: "2"
limits:
nvidia.com/gpu: "2"
mmap:
# Set memory mapping property for whole cluster
mmapEnabled: true
# Set memory-mapped directory path, if you leave mmapDirPath unspecified, the memory-mapped files will be stored in {localStorage.path}/ mmap by default.
mmapDirPath: /mnt/vector/clustersetup_files/
minio:
enabled: false
externalS3:
enabled: true
host: "xx..xx.xxx.xx"
port: "xx"
accessKey: "mykey"
secretKey: "myskey"
useSSL: false
bucketName: "milvusdb"
rootPath: ""
useIAM: false
cloudProvider: "aws"
iamEndpoint: ""`
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the logged post-insert optimize transition in milvus.py and the run completion in serial_runner.py, then compare them with the supplied Helm GPU and externalS3 settings. Done means reproducing or measuring the LAION 100M DISKANN run and documenting what optimize does, expected duration, and how GPU use can be checked.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100