Python GPU IVF training is ~5.8× slower than necessary due to redundant sampling and disk I/O

Problem

Training IVF centroids via the Python GPU path (train_ivf_centroids_on_accelerator) is
extremely slow compared to the Rust CPU path's design. On a 5M×1536d float32 S3-backed
dataset with k=256 and 50 max iterations, IVF training takes ~379 seconds — the vast
majority spent on I/O rather than GPU computation.

Root causes:

1. Double sampling from remote storage: Training data is sampled from S3 twice —
   once to pick k=256 initial centroids (wastefully reading 65536 rows), and again to
   build the KMeans training dataset. Each pass performs 2048 small random takes of 32
   rows, resulting in ~4096 total random S3 read requests.

2. Tiny, unsorted random reads: The old _efficient_sample divides the dataset into
   2048 sequential chunks and picks a small random offset within each. This produces many
   small, scattered read requests that cannot be merged by the object store, causing
   extremely high I/O latency (~226s for a single pass on S3).

3. Disk-based caching between epochs: The second sampling pass writes data to a disk
   IPC file (CachedDataset). Epoch 0 must read all data from S3 first, then epoch 1
   re-reads everything to populate the cache. Only epochs 2+ benefit from the cache —
   but by then most of the time has already been spent.

4. Per-epoch data conversion overhead: Each KMeans epoch reads from disk cache and
   converts Arrow → numpy → Tensor, adding unnecessary CPU overhead for data that fits
   easily in memory (~384 MB).

Observed timing breakdown (5M rows, 1536d, k=256, S3, Apple MPS)

97% of training time is spent on I/O, not GPU computation.

Contrast with Rust CPU path

The Rust CPU path (maybe_sample_training_data) samples once into a single in-memory
FixedSizeListArray and runs all KMeans iterations purely in memory — no disk cache,
no redundant reads, no per-epoch conversion. The Python GPU path should follow the same
strategy.

Expected behavior

• Sample training data once from remote storage using sorted, chunked reads
• Hold training data in memory (~384 MB for 65536×1536 float32)
• Select initial centroids from the in-memory buffer (zero I/O)
• Run all KMeans epochs on an in-memory Tensor (zero disk I/O, zero conversion overhead)
• Target: ~60-70s total (limited by a single S3 sampling pass)

Environment

• Dataset: 5M rows, 1536 dimensions, float32, S3-backed
• IVF params: k=256, sample_rate=256, max_iters=50
• GPU: Apple MPS (also affects CUDA users)
• Lance version: main branch

Start at the Python GPU entry point train_ivf_centroids_on_accelerator and inspect the existing _efficient_sample and CachedDataset flow. Compare it with the Rust path's maybe_sample_training_data design, then verify that sampling occurs once, centroids use the in-memory data, and training avoids disk I/O and repeated conversion while approaching the stated 60–70 second target.