Support distributed optimize of vector index delta segments

Summary

optimize_indices on a large dataset with many delta segments is a single-node operation. On a 1B+ row dataset with ~1000 IVF_RQ deltas a full merge takes hours. The per-partition merge loops over every delta serially inside one process (rust/lance/src/index/vector/builder.rs:1030) and the final write is a single FileWriter per segment, so adding cores or IO does not help beyond one machine.

The building blocks for distributing the merge already exist:

• Dataset::merge_existing_index_segments merges a group of segments into a new uncommitted segment on shared storage (rust/lance/src/index.rs:1376).
• Dataset::commit_existing_index_segments atomically replaces the source deltas with the merged segments in one transaction (rust/lance/src/index.rs:1441).

Two gaps prevent running the merge across multiple workers today.

Gap 1: merged segments cannot be merged again (silent corruption)

merge_partial_vector_auxiliary_files (the kernel behind merge_existing_index_segments) reads every input shard's quantized codes as row-major, while its own output is always written in the query-optimized layout:

• IVF_RQ: it calls the non-idempotent pack_codes unconditionally and writes packed = true.
• IVF_PQ / IVF_HNSW_PQ: it transposes codes unconditionally and writes transposed = true.

Feeding an already-merged segment back into the merge, which is the result of any prior merge level, double packs (RQ) or double transposes (PQ) the codes and silently produces a corrupt index. No error is raised. Recall collapses only at query time (measured ~0.07 vs ~0.495 for a correct IVF_RQ index on the same data). As a result the merge can only run as a single level over row-major shards, so it cannot be fanned out as a tree across workers. SQ and FLAT storage has no layout transform and composes correctly today. Production optimize_indices does not route through this kernel, so this is a latent capability gap rather than a corruption of the normal optimize path.

Reproduction:

1. Build two IVF_RQ (or IVF_PQ) segments over disjoint fragment sets with shared centroids and quantizer model.
2. merge_existing_index_segments([a, b]) producing merged segment m1.
3. merge_existing_index_segments([m1, c]).
4. Query the committed result: recall@100 collapses (~0.07 for RQ) with no error.

Proposed fix: record each input shard's layout flag (RabitQuantizationMetadata.packed, ProductQuantizationMetadata.transposed) while reading shard metadata, thread it to the per-shard read path, and invert the layout back to row-major per partition before the merge applies it once. Failing-first tests assert recall against brute-force ground truth.

Gap 2: no planning API

Callers must hand-roll segment grouping and validation. Proposed: a planning API that validates and partitions an index's delta segments into disjoint, serializable merge tasks, plan_index_segment_merge(index_name, segments_per_task, max_segments_to_merge). Bounding by the newest segments mirrors OptimizeOptions::merge(n) and lets callers fold the deltas without rewriting the large base segment. The existing Python Index dataclass already pickles, so a List[List[Index]] plan can be shipped across process boundaries as-is.

Proposed usage

# plan once
tasks = dataset.plan_index_segment_merge("vec_idx", segments_per_task=32)

# merge each task independently, in parallel
merged = [dataset.merge_existing_index_segments(task) for task in tasks]

# one atomic commit replacing the planned segments
dataset.commit_existing_index_segments("vec_idx", "vector", merged)

Requirements: a shared object store so all workers see the same indices/ directory, and disjoint fragment coverage per task (guaranteed by the planner).

Out of scope

• Distributing the initial shuffle for unindexed data (covered by the existing distributed create path via create_index(fragment_ids=...)).
• Retrain and rebalance distribution, and per-partition output files (format change).
• Documentation for the pre-existing contract that independently built IVF_RQ segments must share one pinned RQ rotation (set via the existing RQBuildParams.rotation parameter). This is a requirement of the current API that tests exercising independently built segments necessarily follow, not new functionality. Only the docs write-up is deferred.

Start with rust/lance/src/index/vector/builder.rs:1030 and rust/lance/src/index.rs:1376,1441, then run the IVF_RQ reproduction described in the issue. Trace merge_partial_vector_auxiliary_files and the existing Index representation before assessing the planning and merge paths. Done means chained merges preserve recall and planned tasks can be merged independently and committed atomically.