Lucene: Look into lower-than-desired IVF-PQ performance on a particular dataset with large dimensions

In the images below, notice that the r8gd-16x index build time performance is the same as GPU. This issue addresses investigating whether or not that is expected and whether or not the GPU heuristics can be improved to have better performance when run with datasets with large dimensions.

---

10M 96d 32fp:

---

---

10M 1536d 32fp (without custom parameter runs):

---

---

10M 1536d 32fp (with custom parameters and higher hnsw-layers runs):

---

GPU instances:
g5: https://instances.vantage.sh/aws/ec2/g5.8xlarge?currency=USD
g6e: https://instances.vantage.sh/aws/ec2/g6e.8xlarge?currency=USD

CPU instances:
c8g: https://instances.vantage.sh/aws/ec2/c8g.48xlarge?currency=USD
r8gd: https://instances.vantage.sh/aws/ec2/r8gd.16xlarge?currency=USD

branches used:
cuvs: main (jars currently registering as 26.08.0)
cuvs-lucene: main
vectorsearch-benchmarks: nvzm123:vb-22__zackm (need to update to use heuristics for IVFPQ parameter setting)

AMI's used:
CPU: amazon/al2023-ami-2023.11.20260514.0-kernel-6.1-arm64
GPU: amazon/Deep Learning OSS Nvidia Driver AMI GPU PyTorch 2.10 (Amazon Linux 2023) 20260425

Note:
out-of-core IVF-PQ doesn't work on cuvs-lucene's main branch yet because https://github.com/rapidsai/cuvs-lucene/pull/141 hasn't been merged yet.

---

Reproduction steps for 10M 96d 32fp:
Use 10M slice from Yandex Deep1B dataset
Build from source using main branches from cuvs and cuvs-lucene
update vectorsearch-benchmarks to use the new heuristics constructors
run tests against configurations and metrics from :
metrics_3a4oFw_GPU__main__g6e-8x_deep-10M-96d_1-seg_1-hnsw-layer.csv
metrics_5YY2w2_CPU__c8g-48x_deep-10M-96d_1-seg.csv
metrics_M0hW5G_GPU__Main__g5-8x_deep-10M-96d_1-seg_1-hnsw-layer.csv
metrics_mbu56b_CPU__g6e-HNSW__deep-10M-96d_1-seg__1.csv
metrics_spKmVV_CPU__r8gd-16x_deep-10M-96d_1-seg.csv

---

Reproduction steps for 10M 1536d 32fp:
Build from source using main branches from cuvs and cuvs-lucene
Cherry-pick in https://github.com/rapidsai/cuvs-lucene/pull/141 to work with latest cuvs-lucene
update vectorsearch-benchmarks to use the new heuristics constructors
run tests against configurations and metrics from :
metrics_5yhvAc_CPU__r8gd-16x_foo-10M-1536d_1-seg.csv
metrics_JysFft_GPU__Main-mayn__g6e-8x_foo-10M-1536d_1-seg_3-hnsw-layers.csv
metrics_QxLVe8_CPU__g6e-HNSW__foo-10M-1536d_1-seg.csv
metrics_ReVEYd_GPU__g6e-8x_foo-10M-1536d_1-seg__main__1-hnswlayer__with-higher-degrees.csv
metrics_s5Xdvs_GPU__main-Mayn__g6e-8x_foo-10M-1536d_1-seg_1-hnsw-layer.csv
metrics_WPMguv_GPU__main-mayn__g6e-8x_foo-10M-1536d_1-seg_4-hnsw-layers.csv
metrics_YHTirw_GPU__Main-mayn__g6e-8x_foo-10M-1536d_1-seg_6-hnsw-layers.csv
metrics_yHX20K_CPU__c8g-48x_foo-10M-1536d_1-seg.csv

---

This issue NVIDIA/cuvs#2466 is only covering slower than desired index build times and the investigation into whether or not heuristics need to be modified to handle large-dimensions (e.g. 1024 - 2048). Must test at 10M. please also test at 5M and 20M.

Issue NVIDIA/cuvs#2473 (https://github.com/NVIDIA/cuvs/issues/2473) is covering the issue regarding the slow search latency

Start with the listed vectorsearch-benchmarks branch and the supplied metrics CSVs, reproducing the 10M 96d and 1536d IVF-PQ index-build comparisons using the referenced cuvs and cuvs-lucene branches. Test 5M, 10M, and 20M datasets, including the 1024–2048 dimension range, and compare CPU/GPU performance to determine whether the heuristics need modification; keep the search-latency investigation in issue #2473.