individual shard_key when adding points
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 304
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 37
Description
when uploading data to qdrant using custom sharding on collection
with all available functions upsert(), upload_records(), upload_collection(), upload_points(), etc
the shard_key_selector operates at global data, so before uploading, i have to batch data by custom shard key
i think it will be more useful to set custom shard key (aka shard_key_selector) at point level, not global
like this
qdrant_client.upsert(
collection_name=COLLECTION_NAME,
points=qdrant_models.Batch(
ids=[str(uuid.uuid4()) for _ in range(len(batch_keys))],
payloads=[dict(office=office_key, tmview_id=batch_key) for batch_key in batch_keys],
vectors=[features_by_office[office_key][batch_key] for batch_key in batch_keys],
shards_keys=[custom_shard_key for custom_shard_key in blablalblab] #this would be useful
),
#shard_key_selector=office_key #not this
)
and ids should be optional, if not set, qdrant server side should set it
otherwise, i am creating lots of random ids on client side and send them, taking size of the final payload size
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing qdrant_client.upsert(), upload_records(), upload_collection(), and upload_points(), along with qdrant_models.Batch, to see where shard_key_selector and ids are defined and validated. Compare the client request models with the server API; done means a decided, tested representation for per-point shard keys and optional IDs, but the issue does not name files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100