qdrant / qdrant/qdrant

Empty snapshots in distributed deployment

Open
#2,972 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
34.7k
Forks
2.7k
Avg merge
1d 18h
Merged PRs (30d)
187

Description

I am faced with the problem of creating snapshots in a distributed qdrant deployment. The collection is very large, but its snapshot is created with a size of 11.8 kB.

Current Behavior

Qdrant is deployed in distributed mode on 5 nodes. The collection was created with 3 shards and a replication factor of 2. I use a load balancer in front of the qdrant cluster. Therefore, requests are randomly distributed across nodes. If the snapshot creation request is directed to a node that does not have any shards of this collection, an empty snapshot of 11 kilobytes is created as a result.
Request:
curl --location --request POST 'http://qdrant:6333/collections/mycollection/snapshots'
Response:
{
"result": {
"name": "mycollection-13542296025311659898-2023-11-10-08-34-23.snapshot",
"creation_time": "2023-11-10T08:34:23",
"size": 11776
},
"status": "ok",
"time": 0.181759348
}

Steps to Reproduce

  1. Set up a distributed Qdrant deployment.
  2. Create a collection with a replication factor less than the number of nodes. Or move the shards in such a way that there are no shards of this collection on some node.
  3. Fill the collection with enough number of vectors.
  4. Send a request to create a snapshot of the collection to a node that does not have shards of this collection.
  5. Check created snapshot.

Expected Behavior

A normal snapshot is created regardless of which node the snapshot creation request came to.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the snapshot request in a five-node distributed deployment with three shards and replication factor two, targeting a node without collection shards. Trace the collection snapshot endpoint and request routing to determine why it creates an 11.8 kB snapshot; done means the same request creates a normal snapshot regardless of the receiving node.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.