504 Gateway Time-out error while performing search operation.

I'm using a qdrant cluster with 3 nodes, Replication factor of 3, and 6 Shards. The collection creation and data ingestion part is working properly via my python script but when I query the collection to retrieve similar chunks I'm facing 504 gateway time-out error.

PFB the error logs:

File "/usr/local/lib/python3.11/site-packages/qdrant_client/qdrant_client.py", line 336, in search
return self._client.search(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/qdrant_client/qdrant_remote.py", line 497, in search
search_result = self.http.points_api.search_points(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/qdrant_client/http/api/points_api.py", line 1388, in search_points
return self._build_for_search_points(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/qdrant_client/http/api/points_api.py", line 636, in build_for_search_points
return self.api_client.request(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/qdrant_client/http/api_client.py", line 74, in request
return self.send(request, type)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/qdrant_client/http/api_client.py", line 97, in send
raise UnexpectedResponse.for_response(response)
qdrant_client.http.exceptions.UnexpectedResponse: Unexpected Response: 504 (Gateway Time-out)
Raw response content:
b'\r\n<title>504 Gateway Time-out</title>\r\n\r\n

504 Gateway Time-out

Initially, this was working when I had 57k points in the collection but after the data increased to 75k, and I upgraded to the latest version (v1.8.0) since I'm facing this issue.

Start with the search call in qdrant_client.py and follow the traceback through qdrant_remote.py, points_api.py, and api_client.py. Compare the 57k-point and 75k-point cases around the v1.8.0 upgrade and determine whether the 504 originates in the client request or the three-node cluster. Done means a reproducible cause and an actionable diagnosis or fix.