Consensus operation timeout issue

I'm running Qdrant vector database engine in Google Kubernetes and it was deployed by official Helm chart with minor modifications in values.yaml. It was running 3 replicas and I decided to increase it to 5. I ran Helm upgrade and it created 2 more replicas and they're all in Running state. When I run a search on existing collections, I get results immediately. However, I can't run create a new collection by using client.recreate_collection method.

My Python client version: 1.0.5
Qdrant docker version version: 1.2.0

Here are my Python logs:

    self.client.recreate_collection(
  File "/opt/conda/default/lib/python3.8/site-packages/qdrant_client/qdrant_client.py", line 1712, in recreate_collection
    self.delete_collection(collection_name)
  File "/opt/conda/default/lib/python3.8/site-packages/qdrant_client/qdrant_client.py", line 1646, in delete_collection
    result: Optional[bool] = self.http.collections_api.delete_collection(
  File "/opt/conda/default/lib/python3.8/site-packages/qdrant_client/http/api/collections_api.py", line 788, in delete_collection
    return self._build_for_delete_collection(
  File "/opt/conda/default/lib/python3.8/site-packages/qdrant_client/http/api/collections_api.py", line 268, in _build_for_delete_collection
    return self.api_client.request(
  File "/opt/conda/default/lib/python3.8/site-packages/qdrant_client/http/api_client.py", line 68, in request
    return self.send(request, type_)
  File "/opt/conda/default/lib/python3.8/site-packages/qdrant_client/http/api_client.py", line 91, in send
    raise UnexpectedResponse.for_response(response)
qdrant_client.http.exceptions.UnexpectedResponse: Unexpected Response: 500 (Internal Server Error)
Raw response content:
b'{"status":{"error":"Service internal error: Waiting for consensus operation commit failed. Timeout set at: 10 seconds"},"time":10.014906989}'

Kubernetes logs:

[2023-11-30T22:27:50.775Z WARN  qdrant::actix::helpers] error processing request: Waiting for consensus operation commit failed. Timeout set at: 10 seconds
[2023-11-30T22:27:50.775Z INFO  actix_web::middleware::logger] 10.40.0.105 "DELETE /collections/3d995dc0510a4d59839d18acdaea4930 HTTP/1.1" 500 133 "-" "python-httpx/0.23.3" 10.015499

How can I fix this issue?

Start with the Helm chart's values.yaml and the deployment changes described in the issue, then reproduce the failure by scaling from three to five replicas and calling the Python client's recreate_collection entry point. Use the Qdrant and Kubernetes logs to determine why the consensus operation does not commit; done means identifying a reproducible cause and documenting or applying a validated fix.