qdrant / qdrant/qdrant-client

Please recommend various option values to upload 100 million data to qdrant.

Open
#130 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
304
Avg merge
2d 9h
Merged PRs (30d)
37

Description

Currently, I am engaged in a job related to natural language processing. The current qdrant can include the embedded vector value through various option values.
But I keep getting the error "broken pipe" when I put in 100 million embedded vectors. The parallel value is 4.
I use the upload_collection function when I put 100 million embedded vector data into qdrant. I would like to get a recommendation from you for various option values of this function.
I would also like to hear an approximate answer about how long it takes to put in 100 million data. I look forward to your good reply.

[vector info]
  1. Number of embedding vectors: 100 million
  2. embedding vector dimension : 768
[hardware spec]

1.cpu core : 10 core
2.cpu : Intel(R) Core(TM) i9-10900X
3.ram Capacity: 251 GiB (Free: 106 GiB)

[upload_collection function Option value]
  1. payload = None
  2. parallel : 4
[collection info]
  1. distance: Cosine
  2. shard_number : 5
  3. memmap_threshold : 1,000,000,000
  4. indexing_threshold : 100,000,000

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file or test is named. Start by reading the Python client's upload_collection entry point and its option documentation, then examine the reported 100-million-vector configuration and broken-pipe failure. Done would require a documented, reproducible recommendation for the options and an evidence-based upload-time estimate.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.