qdrant / qdrant/qdrant-client

qdrant_client.http.exceptions.ResponseHandlingException: [Errno 9] Bad file descriptor

Open
#614 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
304
Avg merge
2d 9h
Merged PRs (30d)
37

Description

I'm making some parallel requests (~50/sec) to a qdrant server in a collection with the following config:

{
  "params": {
    "vectors": {
      "size": 2048,
      "distance": "Cosine",
      "on_disk": true
    },
    "shard_number": 1,
    "replication_factor": 1,
    "write_consistency_factor": 1,
    "on_disk_payload": true
  },
  "hnsw_config": {
    "m": 16,
    "ef_construct": 100,
    "full_scan_threshold": 10000,
    "max_indexing_threads": 0,
    "on_disk": false
  },
  "optimizer_config": {
    "deleted_threshold": 0.2,
    "vacuum_min_vector_number": 1000,
    "default_segment_number": 0,
    "max_segment_size": null,
    "memmap_threshold": null,
    "indexing_threshold": 20000,
    "flush_interval_sec": 5,
    "max_optimization_threads": 1
  },
  "wal_config": {
    "wal_capacity_mb": 32,
    "wal_segments_ahead": 0
  },
  "quantization_config": null
}

And I'm facing a huge delay in responses (4-40 seconds), and sometimes the following error:

search_result = self.client.search(
  File "/usr/local/lib/python3.8/dist-packages/qdrant_client/qdrant_client.py", line 324, in search
    return self._client.search(
  File "/usr/local/lib/python3.8/dist-packages/qdrant_client/qdrant_remote.py", line 454, in search
    search_result = self.http.points_api.search_points(
  File "/usr/local/lib/python3.8/dist-packages/qdrant_client/http/api/points_api.py", line 1197, in search_points
    return self._build_for_search_points(
  File "/usr/local/lib/python3.8/dist-packages/qdrant_client/http/api/points_api.py", line 536, in _build_for_search_points
    return self.api_client.request(
  File "/usr/local/lib/python3.8/dist-packages/qdrant_client/http/api_client.py", line 74, in request
    return self.send(request, type_)
  File "/usr/local/lib/python3.8/dist-packages/qdrant_client/http/api_client.py", line 91, in send
    response = self.middleware(request, self.send_inner)
  File "/usr/local/lib/python3.8/dist-packages/qdrant_client/http/api_client.py", line 200, in __call__
    return call_next(request)
  File "/usr/local/lib/python3.8/dist-packages/qdrant_client/http/api_client.py", line 103, in send_inner
    raise ResponseHandlingException(e)
qdrant_client.http.exceptions.ResponseHandlingException: [Errno 9] Bad file descriptor

i guess that the problem is caused by "on_disk_payload": true, and "on_disk": true, but i'm not sure if this is the cause, and if it is, there is any way to change it without having to recreate and re populate the collection?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with qdrant_client/http/api_client.py, especially send and send_inner, then trace the request from qdrant_client/http/api/points_api.py through search_points. Reproduce the parallel-request case and identify what causes the bad file descriptor and response delays. Done means the cause, relevant configuration or code change, and any safe collection migration path are documented or covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.