How to ensure high availability in Qdrant on Kubernetes during node restarts?
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 34.7k
- Forks
- 2.7k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 187
Description
I'm running Qdrant on Kubernetes (GKE) with 10 Qdrant pods distributed across multiple nodes.
Recently, I ran into an issue: if one of the nodes restarts (for example, due to memory pressure), the Qdrant pods on that node go down for some time and this causes downtime for the entire application.
I haven’t set resource requests and limits yet, but even if I do something like:
resources:
limits:
cpu: 1
memory: 5Gi
requests:
cpu: 2
memory: 10Gi
…it doesn’t seem to help. From what I understand, Qdrant’s architecture expects pods to be stable — if a pod is rescheduled or restarted, it can lead to missing shards or delayed recovery.
Horizontal Pod Autoscaling (HPA) doesn’t help here either, since scaling Qdrant pods dynamically disrupts the shard layout and can cause partial unavailability.
Questions:
- What’s the recommended approach to make a Qdrant deployment resilient to node restarts?
- If I do a Helm upgrade (e.g., updating resources), the pods start rolling one by one. But if my collection has shards distributed across all 10 pods, won't this rolling restart temporarily break the availability of some shards, causing downtime?
Is there a safe upgrade pattern to avoid this?
upgrade pattern to avoid this?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the Qdrant Helm deployment, including resource requests and limits, pod distribution, HPA behavior, and the shard layout described in the issue. Reproduce a node restart and a rolling Helm upgrade, then document a safe deployment pattern that keeps collections available during pod recovery and upgrades.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, helm, kubernetes
- Domain
- cloud, databases, distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100