qdrant_client.http.exceptions.ResponseHandlingException: [Errno 9] Bad file descriptor
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 304
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 37
Description
I'm making some parallel requests (~50/sec) to a qdrant server in a collection with the following config:
{
"params": {
"vectors": {
"size": 2048,
"distance": "Cosine",
"on_disk": true
},
"shard_number": 1,
"replication_factor": 1,
"write_consistency_factor": 1,
"on_disk_payload": true
},
"hnsw_config": {
"m": 16,
"ef_construct": 100,
"full_scan_threshold": 10000,
"max_indexing_threads": 0,
"on_disk": false
},
"optimizer_config": {
"deleted_threshold": 0.2,
"vacuum_min_vector_number": 1000,
"default_segment_number": 0,
"max_segment_size": null,
"memmap_threshold": null,
"indexing_threshold": 20000,
"flush_interval_sec": 5,
"max_optimization_threads": 1
},
"wal_config": {
"wal_capacity_mb": 32,
"wal_segments_ahead": 0
},
"quantization_config": null
}
And I'm facing a huge delay in responses (4-40 seconds), and sometimes the following error:
search_result = self.client.search(
File "/usr/local/lib/python3.8/dist-packages/qdrant_client/qdrant_client.py", line 324, in search
return self._client.search(
File "/usr/local/lib/python3.8/dist-packages/qdrant_client/qdrant_remote.py", line 454, in search
search_result = self.http.points_api.search_points(
File "/usr/local/lib/python3.8/dist-packages/qdrant_client/http/api/points_api.py", line 1197, in search_points
return self._build_for_search_points(
File "/usr/local/lib/python3.8/dist-packages/qdrant_client/http/api/points_api.py", line 536, in _build_for_search_points
return self.api_client.request(
File "/usr/local/lib/python3.8/dist-packages/qdrant_client/http/api_client.py", line 74, in request
return self.send(request, type_)
File "/usr/local/lib/python3.8/dist-packages/qdrant_client/http/api_client.py", line 91, in send
response = self.middleware(request, self.send_inner)
File "/usr/local/lib/python3.8/dist-packages/qdrant_client/http/api_client.py", line 200, in __call__
return call_next(request)
File "/usr/local/lib/python3.8/dist-packages/qdrant_client/http/api_client.py", line 103, in send_inner
raise ResponseHandlingException(e)
qdrant_client.http.exceptions.ResponseHandlingException: [Errno 9] Bad file descriptor
i guess that the problem is caused by "on_disk_payload": true, and "on_disk": true, but i'm not sure if this is the cause, and if it is, there is any way to change it without having to recreate and re populate the collection?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with qdrant_client/http/api_client.py, especially send and send_inner, then trace the request from qdrant_client/http/api/points_api.py through search_points. Reproduce the parallel-request case and identify what causes the bad file descriptor and response delays. Done means the cause, relevant configuration or code change, and any safe collection migration path are documented or covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100