qdrant / qdrant/qdrant

How to ensure high availability in Qdrant on Kubernetes during node restarts?

Open
#6,432 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
34.7k
Forks
2.7k
Avg merge
1d 18h
Merged PRs (30d)
187

Description

I'm running Qdrant on Kubernetes (GKE) with 10 Qdrant pods distributed across multiple nodes.

Recently, I ran into an issue: if one of the nodes restarts (for example, due to memory pressure), the Qdrant pods on that node go down for some time and this causes downtime for the entire application.

I haven’t set resource requests and limits yet, but even if I do something like:

resources:
  limits:
    cpu: 1
    memory: 5Gi
  requests:
    cpu: 2
    memory: 10Gi

…it doesn’t seem to help. From what I understand, Qdrant’s architecture expects pods to be stable — if a pod is rescheduled or restarted, it can lead to missing shards or delayed recovery.

Horizontal Pod Autoscaling (HPA) doesn’t help here either, since scaling Qdrant pods dynamically disrupts the shard layout and can cause partial unavailability.

Questions:

  1. What’s the recommended approach to make a Qdrant deployment resilient to node restarts?
  2. If I do a Helm upgrade (e.g., updating resources), the pods start rolling one by one. But if my collection has shards distributed across all 10 pods, won't this rolling restart temporarily break the availability of some shards, causing downtime?
    Is there a safe upgrade pattern to avoid this?
    upgrade pattern to avoid this?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the Qdrant Helm deployment, including resource requests and limits, pod distribution, HPA behavior, and the shard layout described in the issue. Reproduce a node restart and a rolling Helm upgrade, then document a safe deployment pattern that keeps collections available during pod recovery and upgrades.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, helm, kubernetes
Domain
cloud, databases, distributed-systems, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.