Empty snapshots in distributed deployment
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 34.7k
- Forks
- 2.7k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 187
Description
I am faced with the problem of creating snapshots in a distributed qdrant deployment. The collection is very large, but its snapshot is created with a size of 11.8 kB.
Current Behavior
Qdrant is deployed in distributed mode on 5 nodes. The collection was created with 3 shards and a replication factor of 2. I use a load balancer in front of the qdrant cluster. Therefore, requests are randomly distributed across nodes. If the snapshot creation request is directed to a node that does not have any shards of this collection, an empty snapshot of 11 kilobytes is created as a result.
Request:
curl --location --request POST 'http://qdrant:6333/collections/mycollection/snapshots'
Response:
{
"result": {
"name": "mycollection-13542296025311659898-2023-11-10-08-34-23.snapshot",
"creation_time": "2023-11-10T08:34:23",
"size": 11776
},
"status": "ok",
"time": 0.181759348
}
Steps to Reproduce
- Set up a distributed Qdrant deployment.
- Create a collection with a replication factor less than the number of nodes. Or move the shards in such a way that there are no shards of this collection on some node.
- Fill the collection with enough number of vectors.
- Send a request to create a snapshot of the collection to a node that does not have shards of this collection.
- Check created snapshot.
Expected Behavior
A normal snapshot is created regardless of which node the snapshot creation request came to.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the snapshot request in a five-node distributed deployment with three shards and replication factor two, targeting a node without collection shards. Trace the collection snapshot endpoint and request routing to determine why it creates an 11.8 kB snapshot; done means the same request creates a normal snapshot regardless of the receiving node.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100