[Enhance]: Support the Flat index compute distances through turbo Quantizer
- Dominant language
- C++
- Stars
- 15.9k
- Forks
- 998
- Avg merge
- 4d 7h
- Merged PRs (30d)
- 34
Description
### Affected Component
Flat Searcher, Flat Builder, Flat Streamer
### Current Behavior
The `turbo` module already provides registered quantizers (`Fp32Quantizer`, `Fp16Quantizer`,
`Int8Quantizer`, `Int4Quantizer`, `PqInt8Quantizer`) with own SIMD distance kernels and APIs.
Every distance in the Flat path goes through `FlatDistanceMatrix` / `IndexMetric`
(`distance_matrix().distance<1>(...)`, `row_distance_`, `column_distance_`). Quantized data can
therefore only be searched by going through a Converter/Reformer pair and a matching `IndexMetric`,
which duplicates logic that the turbo quantizers already implement (and better).
### Desired Improvement
Plumb an optional `std::shared_ptr` through the framework and make the Flat
index delegate distance computation to it when present.
### Impact
- Turbo quantizers become usable from the real Flat build/search flow instead of tests only, so
int8/int4/fp16 indexes get the turbo SIMD kernels (AVX2 / AVX‑512 / AVX‑512‑VNNI with scalar
fallback) rather than a Converter + generic `IndexMetric` chain.
- The quantizer choice is persisted in the index meta, so a quantized index is self-describing and
the query side is reconstructed automatically at load time.
- Zero behavior change for existing indexes: without `quantizer_name` in the meta, every code path
falls back to the current metric/distance-matrix logic.
Contributor guide
Assessment
This issue has not been assessed yet.