Please recommend various option values to upload 100 million data to qdrant.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 304
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 37
Description
Currently, I am engaged in a job related to natural language processing. The current qdrant can include the embedded vector value through various option values.
But I keep getting the error "broken pipe" when I put in 100 million embedded vectors. The parallel value is 4.
I use the upload_collection function when I put 100 million embedded vector data into qdrant. I would like to get a recommendation from you for various option values of this function.
I would also like to hear an approximate answer about how long it takes to put in 100 million data. I look forward to your good reply.
[vector info]
- Number of embedding vectors: 100 million
- embedding vector dimension : 768
[hardware spec]
1.cpu core : 10 core
2.cpu : Intel(R) Core(TM) i9-10900X
3.ram Capacity: 251 GiB (Free: 106 GiB)
[upload_collection function Option value]
- payload = None
- parallel : 4
[collection info]
- distance: Cosine
- shard_number : 5
- memmap_threshold : 1,000,000,000
- indexing_threshold : 100,000,000
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file or test is named. Start by reading the Python client's upload_collection entry point and its option documentation, then examine the reported 100-million-vector configuration and broken-pipe failure. Done would require a documented, reproducible recommendation for the options and an evidence-based upload-time estimate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100