VectorDBBench running on Milvus DB giving error - Resource temporarily unavailable

We are running a VectorDB benchmark test on Milvus DB. Below are the specification of the test ran-
Database - Milvus
Case - Search Performance Test (500K Dataset, 1536 Dim)
Index type - DiskANN
search_list - 100 (default)

Sharing the full log here-

#init_bench
2024-09-17 09:59:38,437 | INFO: all configs: [('ALIYUN_OSS_URL', 'assets.zilliz.com.cn/benchmark/'), ('AWS_S3_URL', 'assets.zilliz.com/benchmark/'), ('CONCURRENCY_DURATION', 30), ('CONFIG_LOCAL_DIR', PosixPath('/usr/local/lib/python3.11/site-packages/vectordb_bench/config-files')), ('CUSTOM_CONFIG_DIR', PosixPath('/usr/local/lib/python3.11/site-packages/vectordb_bench/custom/custom_case.json')), ('DATASET_LOCAL_DIR', PosixPath('/tmp/vectordb_bench/dataset')), ('DEFAULT_DATASET_URL', 'assets.zilliz.com/benchmark/'), ('DROP_OLD', True), ('K_DEFAULT', 100), ('LOG_LEVEL', 'INFO'), ('NUM_CONCURRENCY', [1, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100]), ('NUM_PER_BATCH', 5000), ('RESULTS_LOCAL_DIR', PosixPath('/usr/local/lib/python3.11/site-packages/vectordb_bench/results')), ('USE_SHUFFLED_DATA', True)] (main.py:11) (15519)

Collecting usage statistics. To deactivate, set browser.gatherUsageStats to false.

You can now view your Streamlit app in your browser.

Local URL: http://localhost:8501
Network URL: http://10.129.4.79:8501
External URL: http://192.133.242.70:8501

2024-09-17 10:01:34,491 | INFO: generated uuid for the tasks: e2aaa4749e1b41499c63d75622b32a74 (interface.py:66) (15520)
2024-09-17 10:01:34,552 | INFO | DB | CaseType Dataset Filter | task_label (task_runner.py:338)
2024-09-17 10:01:34,552 | INFO | ----------- | ------------ -------------------- ------- | ------- (task_runner.py:338)
2024-09-17 10:01:34,552 | INFO | Milvus | Performance OpenAI-MEDIUM-500K None | test2_nfs_17 (task_runner.py:338)
2024-09-17 10:01:34,552 | INFO: task submitted: id=e2aaa4749e1b41499c63d75622b32a74, test2_nfs_17, case number: 1 (interface.py:231) (15520)
2024-09-17 10:01:34.775 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:01:35,220 | INFO: [1/1] start case: {'label': <CaseLabel.Performance: 2>, 'dataset': {'data': {'name': 'OpenAI', 'size': 500000, 'dim': 1536, 'metric_type': <MetricType.COSINE: 'COSINE'>}}, 'db': 'Milvus'}, drop_old=True (interface.py:164) (15635)
2024-09-17 10:01:35,220 | INFO: Starting run (task_runner.py:100) (15635)
2024-09-17 10:01:35,297 | INFO: Milvus client drop_old collection: VectorDBBenchCollection (milvus.py:45) (15635)
2024-09-17 10:01:35,302 | INFO: Milvus create collection: VectorDBBenchCollection (milvus.py:55) (15635)
2024-09-17 10:01:35,950 | INFO: local dataset root path not exist, creating it: /tmp/vectordb_bench/dataset/openai/openai_medium_500k (data_source.py:119) (15635)
2024-09-17 10:01:35,950 | INFO: Start to downloading files, total count: 3 (data_source.py:135) (15635)
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [02:20<00:00, 46.86s/it]
2024-09-17 10:03:56,531 | INFO: Succeed to download all files, downloaded file count = 3 (data_source.py:140) (15635)
2024-09-17 10:03:56,531 | INFO: Read the entire file into memory: test.parquet (dataset.py:229) (15635)
2024-09-17 10:03:56,665 | INFO: Read the entire file into memory: neighbors.parquet (dataset.py:229) (15635)
2024-09-17 10:03:56,690 | INFO: Start performance case (task_runner.py:158) (15635)
2024-09-17 10:03:56.912 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:03:57,480 | INFO: (SpawnProcess-1:1) Start inserting embeddings in batch 5000 (serial_runner.py:35) (15962)
2024-09-17 10:03:57,480 | INFO: Get iterator for shuffle_train.parquet (dataset.py:247) (15962)
2024-09-17 10:04:37,844 | INFO: (SpawnProcess-1:1) Loaded 100000 embeddings into VectorDB (serial_runner.py:59) (15962)
2024-09-17 10:05:20,985 | INFO: (SpawnProcess-1:1) Loaded 200000 embeddings into VectorDB (serial_runner.py:59) (15962)
2024-09-17 10:06:12,291 | INFO: (SpawnProcess-1:1) Loaded 300000 embeddings into VectorDB (serial_runner.py:59) (15962)
2024-09-17 10:07:06,173 | INFO: (SpawnProcess-1:1) Loaded 400000 embeddings into VectorDB (serial_runner.py:59) (15962)
2024-09-17 10:07:58,594 | INFO: (SpawnProcess-1:1) Loaded 500000 embeddings into VectorDB (serial_runner.py:59) (15962)
2024-09-17 10:07:58,603 | INFO: (SpawnProcess-1:1) Finish loading all dataset into VectorDB, dur=241.12337821099936 (serial_runner.py:61) (15962)
2024-09-17 10:07:59.174 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:08:00,333 | INFO: Milvus post insert before optimize (milvus.py:101) (18207)
2024-09-17 10:18:08,647 | INFO: Milvus optimizing before search (milvus.py:93) (18207)
2024-09-17 10:18:18,257 | INFO: Finish loading the entire dataset into VectorDB, insert_duration=242.15658646000156, optimize_duration=617.8485961160004 load_duration(insert + optimize) = 860.0052 (task_runner.py:167) (15635)
2024-09-17 10:18:18.525 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:18:19,117 | INFO: SpawnProcess-1:3 start search the entire test_data to get recall and latency (serial_runner.py:171) (19339)
2024-09-17 10:18:52,124 | INFO: SpawnProcess-1:3 search entire test_data: cost=25.8579s, queries=1000, avg_recall=0.9896, avg_ndcg=0.9911,avg_latency=0.0259, p99=0.0557 (serial_runner.py:209) (19339)
2024-09-17 10:18:52.526 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:18:52,972 | INFO: Start search 30s in concurrency 1, filters: None (mp_runner.py:99) (15635)
2024-09-17 10:18:53.196 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing
2024-09-17 10:19:29,975 | INFO: Syncing all process and start concurrency search, concurrency=5 (mp_runner.py:108) (15635)
2024-09-17 10:19:59,998 | INFO: SpawnProcess-1:8 search 30s: actual_dur=30.0072s, count=1636, qps in this process: 54.5202 (mp_runner.py:76) (20009)
2024-09-17 10:20:00,006 | INFO: SpawnProcess-1:7 search 30s: actual_dur=30.0141s, count=1626, qps in this process: 54.1745 (mp_runner.py:76) (20006)
2024-09-17 10:20:00,008 | INFO: SpawnProcess-1:11 search 30s: actual_dur=30.0159s, count=1629, qps in this process: 54.2712 (mp_runner.py:76) (20012)
2024-09-17 10:20:00,010 | INFO: SpawnProcess-1:10 search 30s: actual_dur=30.0189s, count=1631, qps in this process: 54.3324 (mp_runner.py:76) (20011)
2024-09-17 10:20:00,011 | INFO: SpawnProcess-1:9 search 30s: actual_dur=30.0186s, count=1630, qps in this process: 54.2997 (mp_runner.py:76) (20010)
2024-09-17 10:20:00,013 | INFO: End search in concurrency 5: dur=30.037750658000732s, total_count=8152, qps=271.3918 (mp_runner.py:120) (15635)
2024-09-17 10:20:00,362 | INFO: Update largest qps with concurrency 5: current max_qps=271.3918 (mp_runner.py:124) (15635)
2024-09-17 10:20:00.585 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:20:01,031 | INFO: Start search 30s in concurrency 10, filters: None (mp_runner.py:99) (15635)
2024-09-17 10:20:11,068 | INFO: Syncing all process and start concurrency search, concurrency=10 (mp_runner.py:108) (15635)
2024-09-17 10:20:41,098 | INFO: SpawnProcess-1:18 search 30s: actual_dur=30.011s, count=885, qps in this process: 29.4892 (mp_runner.py:76) (20990)
2024-09-17 10:20:41,099 | INFO: SpawnProcess-1:13 search 30s: actual_dur=30.0114s, count=887, qps in this process: 29.5554 (mp_runner.py:76) (20983)
2024-09-17 10:20:41,104 | INFO: SpawnProcess-1:15 search 30s: actual_dur=30.0177s, count=895, qps in this process: 29.8157 (mp_runner.py:76) (20987)
2024-09-17 10:20:41,104 | INFO: SpawnProcess-1:17 search 30s: actual_dur=30.0173s, count=892, qps in this process: 29.7162 (mp_runner.py:76) (20989)
2024-09-17 10:20:41,104 | INFO: SpawnProcess-1:21 search 30s: actual_dur=30.0171s, count=891, qps in this process: 29.6831 (mp_runner.py:76) (20993)
2024-09-17 10:20:41,105 | INFO: SpawnProcess-1:22 search 30s: actual_dur=30.0176s, count=892, qps in this process: 29.7159 (mp_runner.py:76) (20994)
2024-09-17 10:20:41,106 | INFO: SpawnProcess-1:20 search 30s: actual_dur=30.0184s, count=890, qps in this process: 29.6485 (mp_runner.py:76) (20992)
2024-09-17 10:20:41,106 | INFO: SpawnProcess-1:19 search 30s: actual_dur=30.0193s, count=894, qps in this process: 29.7808 (mp_runner.py:76) (20991)
2024-09-17 10:20:41,106 | INFO: SpawnProcess-1:16 search 30s: actual_dur=30.0177s, count=885, qps in this process: 29.4826 (mp_runner.py:76) (20988)
2024-09-17 10:20:41,106 | INFO: SpawnProcess-1:14 search 30s: actual_dur=30.0178s, count=890, qps in this process: 29.6491 (mp_runner.py:76) (20986)
2024-09-17 10:20:41,108 | INFO: End search in concurrency 10: dur=30.040207265999925s, total_count=8901, qps=296.3029 (mp_runner.py:120) (15635)
2024-09-17 10:20:41,512 | INFO: Update largest qps with concurrency 10: current max_qps=296.3029 (mp_runner.py:124) (15635)
2024-09-17 10:20:41.734 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:20:42,178 | INFO: Start search 30s in concurrency 15, filters: None (mp_runner.py:99) (15635)
2024-09-17 10:20:42.420 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:20:52,215 | INFO: Syncing all process and start concurrency search, concurrency=15 (mp_runner.py:108) (15635)
2024-09-17 10:21:22,247 | INFO: SpawnProcess-1:31 search 30s: actual_dur=30.0064s, count=578, qps in this process: 19.2626 (mp_runner.py:76) (22862)
2024-09-17 10:21:22,248 | INFO: SpawnProcess-1:27 search 30s: actual_dur=30.005s, count=578, qps in this process: 19.2635 (mp_runner.py:76) (22858)
2024-09-17 10:21:22,249 | INFO: SpawnProcess-1:38 search 30s: actual_dur=30.009s, count=577, qps in this process: 19.2276 (mp_runner.py:76) (22869)
2024-09-17 10:21:22,250 | INFO: SpawnProcess-1:34 search 30s: actual_dur=30.0074s, count=581, qps in this process: 19.3619 (mp_runner.py:76) (22865)
2024-09-17 10:21:22,253 | INFO: SpawnProcess-1:36 search 30s: actual_dur=30.0156s, count=574, qps in this process: 19.1234 (mp_runner.py:76) (22867)
2024-09-17 10:21:22,254 | INFO: SpawnProcess-1:24 search 30s: actual_dur=30.0121s, count=576, qps in this process: 19.1923 (mp_runner.py:76) (22853)
2024-09-17 10:21:22,255 | INFO: SpawnProcess-1:29 search 30s: actual_dur=30.012s, count=577, qps in this process: 19.2256 (mp_runner.py:76) (22860)
2024-09-17 10:21:22,256 | INFO: SpawnProcess-1:33 search 30s: actual_dur=30.0164s, count=581, qps in this process: 19.3561 (mp_runner.py:76) (22864)
2024-09-17 10:21:22,260 | INFO: SpawnProcess-1:32 search 30s: actual_dur=30.0173s, count=584, qps in this process: 19.4554 (mp_runner.py:76) (22863)
2024-09-17 10:21:22,261 | INFO: SpawnProcess-1:28 search 30s: actual_dur=30.0191s, count=587, qps in this process: 19.5542 (mp_runner.py:76) (22859)
2024-09-17 10:21:22,263 | INFO: SpawnProcess-1:25 search 30s: actual_dur=30.0217s, count=583, qps in this process: 19.4193 (mp_runner.py:76) (22856)
2024-09-17 10:21:22,264 | INFO: SpawnProcess-1:26 search 30s: actual_dur=30.0245s, count=576, qps in this process: 19.1843 (mp_runner.py:76) (22857)
2024-09-17 10:21:22,264 | INFO: SpawnProcess-1:37 search 30s: actual_dur=30.0236s, count=587, qps in this process: 19.5513 (mp_runner.py:76) (22868)
2024-09-17 10:21:22,264 | INFO: SpawnProcess-1:35 search 30s: actual_dur=30.0258s, count=581, qps in this process: 19.35 (mp_runner.py:76) (22866)
2024-09-17 10:21:22,266 | INFO: SpawnProcess-1:30 search 30s: actual_dur=30.0242s, count=576, qps in this process: 19.1845 (mp_runner.py:76) (22861)
2024-09-17 10:21:22,267 | INFO: End search in concurrency 15: dur=30.05192263400022s, total_count=8696, qps=289.3658 (mp_runner.py:120) (15635)
2024-09-17 10:21:22.933 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:21:23,382 | INFO: Start search 30s in concurrency 20, filters: None (mp_runner.py:99) (15635)
2024-09-17 10:21:23.624 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:21:23.705 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing
2024-09-17 10:21:33,434 | INFO: Syncing all process and start concurrency search, concurrency=20 (mp_runner.py:108) (15635)
2024-09-17 10:22:03,471 | INFO: SpawnProcess-1:40 search 30s: actual_dur=30.0123s, count=437, qps in this process: 14.5607 (mp_runner.py:76) (25609)
2024-09-17 10:22:03,472 | INFO: SpawnProcess-1:50 search 30s: actual_dur=30.0141s, count=435, qps in this process: 14.4932 (mp_runner.py:76) (25621)
2024-09-17 10:22:03,474 | INFO: SpawnProcess-1:54 search 30s: actual_dur=30.0157s, count=433, qps in this process: 14.4258 (mp_runner.py:76) (25625)
2024-09-17 10:22:03,474 | INFO: SpawnProcess-1:59 search 30s: actual_dur=30.0168s, count=435, qps in this process: 14.4919 (mp_runner.py:76) (25630)
2024-09-17 10:22:03,475 | INFO: SpawnProcess-1:44 search 30s: actual_dur=30.0151s, count=438, qps in this process: 14.5927 (mp_runner.py:76) (25615)
2024-09-17 10:22:03,476 | INFO: SpawnProcess-1:45 search 30s: actual_dur=30.018s, count=435, qps in this process: 14.4913 (mp_runner.py:76) (25616)
2024-09-17 10:22:03,478 | INFO: SpawnProcess-1:58 search 30s: actual_dur=30.0182s, count=433, qps in this process: 14.4246 (mp_runner.py:76) (25629)
2024-09-17 10:22:03,480 | INFO: SpawnProcess-1:49 search 30s: actual_dur=30.0205s, count=436, qps in this process: 14.5234 (mp_runner.py:76) (25620)
2024-09-17 10:22:03,482 | INFO: SpawnProcess-1:51 search 30s: actual_dur=30.0238s, count=440, qps in this process: 14.655 (mp_runner.py:76) (25622)
2024-09-17 10:22:03,483 | INFO: SpawnProcess-1:57 search 30s: actual_dur=30.0271s, count=439, qps in this process: 14.6201 (mp_runner.py:76) (25628)
2024-09-17 10:22:03,486 | INFO: SpawnProcess-1:48 search 30s: actual_dur=30.0287s, count=439, qps in this process: 14.6193 (mp_runner.py:76) (25619)
2024-09-17 10:22:03,488 | INFO: SpawnProcess-1:46 search 30s: actual_dur=30.0304s, count=435, qps in this process: 14.4853 (mp_runner.py:76) (25617)
2024-09-17 10:22:03,489 | INFO: SpawnProcess-1:53 search 30s: actual_dur=30.0343s, count=435, qps in this process: 14.4834 (mp_runner.py:76) (25624)
2024-09-17 10:22:03,493 | INFO: SpawnProcess-1:43 search 30s: actual_dur=30.0324s, count=440, qps in this process: 14.6508 (mp_runner.py:76) (25614)
2024-09-17 10:22:03,494 | INFO: SpawnProcess-1:52 search 30s: actual_dur=30.0388s, count=438, qps in this process: 14.5811 (mp_runner.py:76) (25623)
2024-09-17 10:22:03,498 | INFO: SpawnProcess-1:47 search 30s: actual_dur=30.0409s, count=434, qps in this process: 14.447 (mp_runner.py:76) (25618)
2024-09-17 10:22:03,498 | INFO: SpawnProcess-1:55 search 30s: actual_dur=30.0375s, count=433, qps in this process: 14.4153 (mp_runner.py:76) (25626)
2024-09-17 10:22:03,498 | INFO: SpawnProcess-1:41 search 30s: actual_dur=30.0386s, count=439, qps in this process: 14.6145 (mp_runner.py:76) (25612)
2024-09-17 10:22:03,499 | INFO: SpawnProcess-1:42 search 30s: actual_dur=30.0421s, count=432, qps in this process: 14.3798 (mp_runner.py:76) (25613)
2024-09-17 10:22:03,499 | INFO: SpawnProcess-1:56 search 30s: actual_dur=30.0395s, count=437, qps in this process: 14.5475 (mp_runner.py:76) (25627)
2024-09-17 10:22:03,501 | INFO: End search in concurrency 20: dur=30.067121991000022s, total_count=8723, qps=290.1176 (mp_runner.py:120) (15635)
2024-09-17 10:22:04.157 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:22:04,613 | INFO: Start search 30s in concurrency 25, filters: None (mp_runner.py:99) (15635)
2024-09-17 10:22:04.868 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:22:04.868 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-09-17 10:22:14,655 | INFO: Syncing all process and start concurrency search, concurrency=25 (mp_runner.py:108) (15635)
OpenBLAS blas_thread_init: pthread_create failed for thread 22 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4194304 current, 4194304 max
OpenBLAS blas_thread_init: pthread_create failed for thread 23 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4194304 current, 4194304 max

Usingpymilvus, we validated that the db connection, creation of collections and indexing are successful.