feature: quantized refinement results cache for high-bit RaBitQ/SQ

Problem

Lance vector search currently offers two quality tiers.

1. Coarse ANN search over compressed index data is fast but approximate, so ranking is limited by the precision of the compressed representation.
2. Exact refinement re-ranks candidates against their original vectors, but those vectors are read from the base dataset. When the dataset is in object storage, those reads can dominate the query, so users pay a large I/O cost for better ordering.

There is no middle option: users either accept the coarse ranking or pay for base-table vector reads.

Motivation

The index already stores higher-precision compressed representations — RQ4/RQ8 (multi-bit RaBitQ) and SQ8 — that are significantly more accurate than the coarse one-bit search yet far cheaper to read than raw vectors. These are currently not used to re-rank candidates, leaving accuracy on the table without requiring base-table reads.

Proposed approach

Add an intermediate refinement tier that re-scores a candidate pool from the higher-bit quantized data already present in the vector index, avoiding base-table reads:

coarse ANN search -> quantized re-score (RQ4/RQ8/SQ8) -> optional exact re-score (raw vectors)

• The tier should only be reused when reuse is provably safe — candidates must stay valid across dataset changes, index changes, and recompaction. Any case that cannot be proven safe must fall back to the existing path.
• Exact refinement stays available and unchanged; this is a cheaper middle step, not a replacement for "exact."
• The feature should introduce no new persistent file-format contract and no behavior change when disabled.

I found a branch with this idea partially implemented. It's old but it can be used as a inspiration for the implementation.

Start by reviewing the existing vector-index search and exact-refinement paths, then compare them with the partially implemented yang/vector-results-cache branch linked in the issue. Define the safe reuse conditions for dataset, index, and recompaction changes, and verify that disabled behavior and exact refinement remain unchanged without introducing a persistent file-format contract.