Slow ANN search on very large distributed IVF_PQ index with many deltas

Summary

We are testing Lance / lance-ray on a very large single-table dataset and are seeing unexpectedly slow ANN search performance on an already-built distributed IVF_PQ vector index.

A single vector search with k=10, minimum_nprobes=1, and maximum_nprobes=2 takes about 200 seconds according to scanner.analyze_plan().

The most expensive operator appears to be ANNIvfPartition, which spends ~200s and reports:

• deltas=500
• deltas_searched=500
• indices_loaded=499
• partitions_ranked=25.00 M
• find_partitions_elapsed=2.95s

This looks like query latency is dominated by searching across many index deltas / distributed index fragments, even though maximum_nprobes is very small.

Environment / Deployment

• Storage: HDFS-compatible distributed filesystem
• Execution mode: Ray / KubeRay-style distributed job
• Vector index built through lance_ray.create_index(...)
• Query executed from the Python driver with lance.LanceDataset(...).scanner(...).analyze_plan()

Dataset

Single Lance table:

• Rows: 50,000,000,000 (50B)
• Approx raw vector + payload size: ~500TB scale
• Fragment count: around 1,000,000 fragments in earlier metadata output
• Data was written in large chunks
• Data columns:
  • id: int64
  • category_id: int64
  • score: float64
  • name: string
  • email: string
  • is_active: bool
  • created_at: timestamp ns
  • embedding: fixed-size list / vector, 256-dimensional float32
  • payload: fixed-size binary padding column used to reach target data size

Data generation summary:

ROW_COUNT = 50_000_000_000
VECTOR_DIM = 256
PAYLOAD_BYTES_PER_ROW = 9_600
CHUNK_ROWS = 50_000_000
min_rows_per_file = 65_536
max_rows_per_file = 524_288

Vector Index

Logical index name:

embedding_ivf_pq

The index was built with lance-ray using roughly the following parameters:

create_index(
    dataset_uri,
    column="embedding",
    index_type="IVF_PQ",
    name="embedding_ivf_pq",
    replace=False,
    num_workers=500,
    block_size=512 * 1024,
    ray_remote_args={"num_cpus": 1.0},
    metric="l2",
    num_partitions=50_000,
    num_sub_vectors=16,
    sample_rate=16,
)

dataset.list_indices() returned 500 index metadata records, all for the same logical index name:

{
  "index_count": 500,
  "index_names": ["embedding_ivf_pq"],
  "vector_index_names": ["embedding_ivf_pq"]
}

My understanding is that these 500 records correspond to distributed index fragments / deltas from the lance-ray distributed build.

Query

The query is executed with scanner.analyze_plan():

q = query_vector  # deterministic 256-dim float32 vector

scanner = ds.scanner(
    nearest={
        "column": "embedding",
        "k": 10,
        "q": q,
        "minimum_nprobes": 1,
        "maximum_nprobes": 2,
    },
    fast_search=True,
    columns=["id", "category_id", "score", "_distance"],
    disable_scoring_autoprojection=True,
)

plan = scanner.analyze_plan()

Observed Result

One analyze_plan() run produced:

AnalyzeExec verbose=true, elapsed=200.256483169s, metrics=[]
  TracedExec, elapsed=200.256483169s, metrics=[]
    ProjectionExec: elapsed=200.256483169s, expr=[id@2 as id, category_id@3 as category_id, score@4 as score, _distance@0 as _distance, _rowid@1 as _rowid], metrics=[output_rows=10, elapsed_compute=9.67µs, output_bytes=192.2 KB, output_batches=1, expr_0_eval_time=1.30µs, expr_1_eval_time=141ns, expr_2_eval_time=144ns, expr_3_eval_time=152ns, expr_4_eval_time=416ns]
      Take: elapsed=200.206890153s, columns="_distance, _rowid, (id), (category_id), (score)", metrics=[output_rows=10, elapsed_compute=23.60ms, output_bytes=0.0 B, output_batches=0, batches_processed=1, bytes_read=0, iops=0, requests=0]
        CoalesceBatchesExec: elapsed=200.206890153s, target_batch_size=16384, metrics=[output_rows=10, elapsed_compute=7.35µs, output_bytes=192.0 KB, output_batches=1]
          SortExec: elapsed=200.206814315s, TopK(fetch=10), expr=[_distance@0 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false], filter=[_distance@0 < 20.844498 OR _distance@0 = 20.844498 AND _rowid@1 < 46372761911816], metrics=[output_rows=10, elapsed_compute=13.98ms, output_bytes=120.0 B, output_batches=1, row_replacements=66]
            ANNSubIndex: elapsed=200.206814315s, name=embedding_ivf_pq, k=10, deltas=500, metric=L2, metrics=[output_rows=9.99 K, elapsed_compute=200.21s, output_bytes=0.0 B, output_batches=0, index_comparisons=6.06 M, indices_loaded=428, partitions_searched=1.00 K, parts_loaded=1.00 K]
              ANNIvfPartition: elapsed=200.150695069s, uuid=<redacted>, minimum_nprobes=1, maximum_nprobes=Some(2), deltas=500, metrics=[output_rows=500, elapsed_compute=200.15s, output_bytes=0.0 B, output_batches=0, deltas_searched=500, index_comparisons=0, indices_loaded=499, partitions_ranked=25.00 M, parts_loaded=0, find_partitions_elapsed=2.95s]

Questions

1. Is it expected that a search with maximum_nprobes=2 still reports partitions_searched=1.00 K and partitions_ranked=25.00 M when the distributed index has deltas=500?
2. Is the query effectively probing up to maximum_nprobes partitions per delta, leading to approximately 500 * 2 = 1000 partitions searched?
3. Is there a recommended way to compact / merge / optimize these 500 vector index deltas into fewer searchable units to avoid the query touching hundreds of index fragments?
4. Are there tuning parameters for large distributed IVF_PQ indexes that reduce driver memory usage and latency during ANN search?
5. Would a different distributed index build strategy be recommended for a 50B-row / ~500TB-scale dataset?

Expected Behavior

With a prebuilt IVF_PQ index and maximum_nprobes=2, we expected top-k search latency to be much lower than 200 seconds, or at least to avoid scanning/ranking across all 500 deltas for every query.

Any guidance on whether this is expected behavior, a tuning issue, or a possible optimization opportunity would be appreciated.

Start with the lance_ray.create_index(...) entry point and the Python LanceDataset.scanner(...).analyze_plan() query described in the issue. Inspect how the reported ANNSubIndex and ANNIvfPartition metrics account for 500 deltas, then determine whether compaction or query tuning is supported. Done means documenting the expected probing behavior and a recommended strategy for reducing latency at this scale.