Define storage & serialization for quantization types MinMax.
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 1.9k
- Forks
- 454
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 35
Description
Define storage & serialization for quantization types (Blanket Issue)
Description
As we expand multi-vector support beyond standard floating-point types, we need a robust storage and serialization strategy for quantized data. Different quantization schemes (e.g., MinMax, Scalar Quantization) require specific metadata—such as scaling factors, zero-points, or offsets—to be stored alongside the compressed vector representations and routed to the Reduce abstraction.
Tasks
- Define a standardized interface for serializing and deserializing quantized multi-vector data and its associated metadata.
- Implement the storage layout and serialization logic specifically for MinMax quantization (handling scales and offsets).
- Ensure the deserialized metadata feeds efficiently into the newly abstracted
Reducepipeline. - Blanket Task: Create sub-issues for other required quantization types as needed (e.g., specific SQ variants).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by mapping the existing quantization types and the newly abstracted Reduce pipeline, then break this blanket issue into focused sub-issues. Define the serialization interface and MinMax storage requirements before implementing them. Done means MinMax metadata and compressed vectors can be serialized, deserialized, and passed correctly into Reduce, with coverage for the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100