Distributed deployment cluster with a single dead shard fails to respond to queries

Running a distributed deployment of qdrant on kubernetes, single collection with replication factor set to 2. One shard failed and the cluster fails to respond to a query. I thought the loss of a single shard in this configuration shouldn't be a problem.

Current Behavior

The cluster fails to run a query

[38;20m2024-02-11 17:28:39 INFO Received input: session_id=62049 query='xxxx xxxxx xxxxx' team_id=1866 file_id=[3074553] sitemap_id=None env_name='xxxxxx'[0m
INFO:     100.100.29.236:39890 - "POST /qdrant/conversation HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  
  File "/home/searchie/.local/lib/python3.10/site-packages/qdrant_client/http/api_client.py", line 97, in send
    raise UnexpectedResponse.for_response(response)
qdrant_client.http.exceptions.UnexpectedResponse: Unexpected Response: 500 (Internal Server Error)
Raw response content:
b'{"status":{"error":"Service internal error: The replica set for shard 3 on peer 6358360577973509 does not have enough active replicas"},"time":0.000686151}'

Steps to Reproduce

1. Have one shard of a distributed deployment cluster, with replication factor = 2, fail
2. Run a query

Expected Behavior

Qdrant continues to index new vectors and respond to search requests.

Possible Solution

Context (Environment)

3 node cluster of qdrant 1.7.3 on kubernetes installed with helm chart qdrant-0.7.5. Each node is aws ec2 r6a.2xlarge with 8vCPU and 64 GB RAM, 100 GB gp3 EBS volume.

{
  "result": {
    "status": "enabled",
    "peer_id": 3141207761334255,
    "peers": {
      "6358360577973509": {
        "uri": "http://qdrant-1.qdrant-headless:6335/"
      },
      "419724648802618": {
        "uri": "http://qdrant-2.qdrant-headless:6335/"
      },
      "3141207761334255": {
        "uri": "http://qdrant-0.qdrant-headless:6335/"
      }
    },
    "raft_info": {
      "term": 364,
      "commit": 14441,
      "pending_operations": 0,
      "leader": 6358360577973509,
      "role": "Follower",
      "is_voter": true
    },
    "consensus_thread_status": {
      "consensus_thread_status": "working",
      "last_update": "2024-02-12T12:40:42.218687606Z"
    },
    "message_send_failures": {}
  },
  "status": "ok",
  "time": 0.000006651
}

{
  "result": {
    "status": "green",
    "optimizer_status": "ok",
    "vectors_count": 14122727,
    "indexed_vectors_count": 14105677,
    "points_count": 14122470,
    "segments_count": 48,
    "config": {
      "params": {
        "vectors": {
          "size": 768,
          "distance": "Cosine",
          "on_disk": true
        },
        "shard_number": 6,
        "replication_factor": 2,
        "write_consistency_factor": 1,
        "on_disk_payload": true
      },
      "hnsw_config": {
        "m": 16,
        "ef_construct": 100,
        "full_scan_threshold": 10000,
        "max_indexing_threads": 0,
        "on_disk": true
      },
      "optimizer_config": {
        "deleted_threshold": 0.2,
        "vacuum_min_vector_number": 1000,
        "default_segment_number": 0,
        "max_segment_size": null,
        "memmap_threshold": null,
        "indexing_threshold": 20000,
        "flush_interval_sec": 5,
        "max_optimization_threads": 1
      },
      "wal_config": {
        "wal_capacity_mb": 32,
        "wal_segments_ahead": 0
      },
      "quantization_config": null
    },
    "payload_schema": {
      "metadata.model_id": {
        "data_type": "integer",
        "points": 14122470
      },
      "metadata.team_id": {
        "data_type": "integer",
        "points": 14122470
      }
    }
  },
  "status": "ok",
  "time": 0.001136403
}

{
  "result": {
    "peer_id": 6358360577973509,
    "shard_count": 6,
    "local_shards": [
      {
        "shard_id": 0,
        "points_count": 2341907,
        "state": "Active"
      },
      {
        "shard_id": 2,
        "points_count": 1944223,
        "state": "Active"
      },
      {
        "shard_id": 3,
        "points_count": 2434372,
        "state": "Dead"
      },
      {
        "shard_id": 5,
        "points_count": 2756990,
        "state": "Active"
      }
    ],
    "remote_shards": [
      {
        "shard_id": 0,
        "peer_id": 419724648802618,
        "state": "Active"
      },
      {
        "shard_id": 1,
        "peer_id": 3141207761334255,
        "state": "Active"
      },
      {
        "shard_id": 1,
        "peer_id": 419724648802618,
        "state": "Active"
      },
      {
        "shard_id": 2,
        "peer_id": 3141207761334255,
        "state": "Active"
      },
      {
        "shard_id": 3,
        "peer_id": 419724648802618,
        "state": "Active"
      },
      {
        "shard_id": 4,
        "peer_id": 3141207761334255,
        "state": "Active"
      },
      {
        "shard_id": 4,
        "peer_id": 419724648802618,
        "state": "Active"
      },
      {
        "shard_id": 5,
        "peer_id": 3141207761334255,
        "state": "Active"
      }
    ],
    "shard_transfers": []
  },
  "status": "ok",
  "time": 0.000043171
}

The cluster writes log like this every 10 seconds

qdrant-1 qdrant 2024-02-12T13:18:49.889916Z  WARN storage::content_manager::consensus_manager: Failed to apply collection meta operation entry with user error: Wrong input: Cannot deactivate the last active replica 419724648802618 of shard 3    
qdrant-2 qdrant 2024-02-12T13:18:49.894157Z  WARN storage::content_manager::consensus_manager: Failed to apply collection meta operation entry with user error: Wrong input: Cannot deactivate the last active replica 419724648802618 of shard 3    
qdrant-0 qdrant 2024-02-12T13:18:49.894543Z  WARN storage::content_manager::consensus_manager: Failed to apply collection meta operation entry with user error: Wrong input: Cannot deactivate the last active replica 419724648802618 of shard 3

Detailed Description

Start by reproducing the failure on Qdrant 1.7.3 with the stated three-node Kubernetes cluster, six shards, and replication factor 2. Inspect the distributed shard replica handling around a Dead shard and the repeated “Cannot deactivate the last active replica” warning; done means queries and new vector indexing continue after one replica fails.