HNSW search prefetch (do_prefetch/look-ahead) shows no benefit at 768 dimensions

Environment

• lance version: 8.0.0
• CPU arch: x86_64
• Index type: HNSW_FLAT (HNSW + flat, unquantized storage)

Description

Index type is HNSW_FLAT. The query-time call path is:

 HNSW::search_basic
   -> search_inner
     -> run_search
       -> beam_search (bottom-level search)
         -> process_neighbors_with_look_ahead
           -> DistCalculator::prefetch(id)
             -> do_prefetch  (lance-index/src/vector/utils.rs)

While iterating the current node's neighbor list, process_neighbors_with_look_ahead calls dist_calc.prefetch(id) ahead of time for the neighbor look_ahead steps in the future, which
triggers do_prefetch to bring that neighbor's vector data into cache. The intent is to hide memory-access latency during graph traversal.

However, for 768-dimensional f32 vectors under an HNSW_FLAT index, this look-ahead prefetch path does not show any measurable positive benefit during vector search.

Suspected cause

Under flat storage, prefetch(id) fetches the full byte slice of the vector for that id and passes it to do_prefetch. A 768-dim f32 vector is 768 * 4 = 3072 bytes = 48 cache lines, and
do_prefetch issues one _mm_prefetch(_MM_HINT_T0) instruction per 64-byte cache line across that range. At this vector size:

• process_neighbors_with_look_ahead triggers one prefetch call per neighbor processed, and each call issues 48 instructions — this issue overhead itself may already be comparable to, or
  exceed, the memory latency it's meant to hide.
• The current look-ahead distance may have been tuned for smaller or quantized vector representations (e.g. PQ/SQ codes). For a ~3KB vector, that distance may not be far enough ahead —
  the prefetched data may not be ready by the time it's actually needed, making the prefetch ineffective while still consuming memory bandwidth.

This suggests the current look-ahead prefetch strategy under HNSW_FLAT may be tuned for smaller vector representations and does not generalize well to large, unquantized vectors at higher
dimensions.

Questions

• Should the look-ahead distance in process_neighbors_with_look_ahead be adjusted dynamically based on vector byte size/dimension, rather than being a fixed value?
• Has this been benchmarked for large-dimension HNSW_FLAT (768+ dims, no PQ/SQ quantization)? If not, would it be worth adding a benefit/regression benchmark at this dimension to the
  existing hnsw bench suite (benches/hnsw.rs) in lance-index?
• Currently prefetch_distance is set at index build time (via HnswBuildParams) — is there a way to disable prefetch at query time (e.g. pass None)? If it can only be fixed at build time,
  would it be useful to expose a more flexible switch (e.g. a query-time override) for cases like this, where prefetch is found to have no benefit and users want to turn it off without
  rebuilding the index?

Start with the HNSW::search_basic call path and inspect process_neighbors_with_look_ahead and do_prefetch in lance-index/src/vector/utils.rs. Review benches/hnsw.rs and HnswBuildParams, then benchmark 768-dimensional HNSW_FLAT searches with and without prefetch. Done means documenting the measured result and determining whether a query-time disable or adjusted look-ahead setting is warranted.