microsoft / microsoft/DiskANN

Define storage & serialization for quantization types MinMax.

Open
#992 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
1.9k
Forks
454
Avg merge
3d 22h
Merged PRs (30d)
35

Description

Define storage & serialization for quantization types (Blanket Issue)

Description

As we expand multi-vector support beyond standard floating-point types, we need a robust storage and serialization strategy for quantized data. Different quantization schemes (e.g., MinMax, Scalar Quantization) require specific metadata—such as scaling factors, zero-points, or offsets—to be stored alongside the compressed vector representations and routed to the Reduce abstraction.

Tasks

  • Define a standardized interface for serializing and deserializing quantized multi-vector data and its associated metadata.
  • Implement the storage layout and serialization logic specifically for MinMax quantization (handling scales and offsets).
  • Ensure the deserialized metadata feeds efficiently into the newly abstracted Reduce pipeline.
  • Blanket Task: Create sub-issues for other required quantization types as needed (e.g., specific SQ variants).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by mapping the existing quantization types and the newly abstracted Reduce pipeline, then break this blanket issue into focused sub-issues. Define the serialization interface and MinMax storage requirements before implementing them. Done means MinMax metadata and compressed vectors can be serialized, deserialized, and passed correctly into Reduce, with coverage for the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.