zilliztech / zilliztech/VectorDBBench
TypeError: cannot pickle '_thread.RLock' object
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 438
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 1
Description
Hi I'm a newbie here, but when I kicked off this benchmark with Search Performance Test, I received the following error immediately. I tried to add some print statements, but couldn't zero in on the error. Appreciate any help. Thank you in advance!
2025-04-04 18:55:32,090 | INFO: Read the entire file into memory: test.parquet (dataset.py:247) (15253)
2025-04-04 18:55:32,116 | INFO: Read the entire file into memory: neighbors.parquet (dataset.py:247) (15253)
2025-04-04 18:55:32,162 | INFO: Start performance case (task_runner.py:147) (15253)
2025-04-04 18:55:32,162 | INFO: dropping old ------------- (task_runner.py:151) (15253)
2025-04-04 18:55:32,162 | INFO: task stage load ------------- (task_runner.py:153) (15253)
2025-04-04 18:55:32,202 | WARNING: VectorDB load dataset error: cannot pickle '_thread.RLock' object (serial_runner.py:141) (15253)
2025-04-04 18:55:33,670 | WARNING: Failed to run performance case, reason = cannot pickle '_thread.RLock' object (task_runner.py:185) (15253)
Traceback (most recent call last):
File "/home/ubuntu/ZillizVectorDBBench/vectordb_bench/backend/task_runner.py", line 154, in _run_perf_case
_, load_dur = self._load_train_data()
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/ZillizVectorDBBench/vectordb_bench/backend/utils.py", line 43, in inner
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/ZillizVectorDBBench/vectordb_bench/backend/task_runner.py", line 204, in _load_train_data
raise e from None
File "/home/ubuntu/ZillizVectorDBBench/vectordb_bench/backend/task_runner.py", line 202, in _load_train_data
runner.run()
File "/home/ubuntu/ZillizVectorDBBench/vectordb_bench/backend/runner/serial_runner.py", line 182, in run
count, dur = self._insert_all_batches()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/ZillizVectorDBBench/vectordb_bench/backend/utils.py", line 43, in inner
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/ZillizVectorDBBench/vectordb_bench/backend/runner/serial_runner.py", line 142, in _insert_all_batches
raise e from e
File "/home/ubuntu/ZillizVectorDBBench/vectordb_bench/backend/runner/serial_runner.py", line 133, in _insert_all_batches
count = future.result(timeout=self.timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/usr/lib/python3.12/multiprocessing/queues.py", line 264, in _feed
obj = _ForkingPickler.dumps(obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
TypeError: cannot pickle '_thread.RLock' object
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in vectordb_bench/backend/runner/serial_runner.py, especially _insert_all_batches, then follow the call path through task_runner.py and the Search Performance Test benchmark. Reproduce the load stage and trace the multiprocessing failure; done means the benchmark completes without the reported RLock pickling error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, performance, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100