Lucene: IVF_PQ not performing up to expectations

Note: This may be due to misconfigured settings.

Machine details:
GPU ec2 instance = g5.8x: A10G with 24 GB VRAM, 32 vCPUs, 128 GB (cpu) RAM, and using NVMe storage, w/ network perf of 25 gigabit
CPU ec2 instance = r8gd.8x: 32 vCPUs, 256 GB (cpu) RAM, and using NVMe storage, w/ network perf of 15 gigabit

dataset size: 10M 32fp 1536d

GPU index-build algorithm: CAGRA_HNSW with IVF_PQ (for out-of-core index-building)
CPU index-build algorithm: LUCENE_HNSW

Please DM for access to full logs:
CPU logs: https://drive.google.com/drive/folders/1y6bv_zPgbDyAX_ndeEcRTUhiRPl7VIvv?usp=drive_link

GPU logs: https://drive.google.com/drive/folders/1DJGAgETGEen8eFsQVfEW3q-dEfxwanFG?usp=drive_link

logs include: metrics.csv files compiled from results.json files, benchmark logs, plots, original results.json files

some GPU IVF-PQ metrics:

"metrics" : {
    "CAGRA_HNSW-index-size" : 59.08250856399536,
    "CAGRA_HNSW-indexing-time" : 1642971,
    "CAGRA_HNSW-mean-latency" : 3599.6695278969955,
    "CAGRA_HNSW-mean-retrieval-latency" : 4.23175965665236,
    "CAGRA_HNSW-query-throughput" : 0.2808124240402393,
    "CAGRA_HNSW-query-time" : 890274,
    "CAGRA_HNSW-recall-accuracy" : 79.2038626609442,
    "CAGRA_HNSW-segment-count" : 25
  }

some CPU HNSW metrics:

"metrics" : {
    "LUCENE_HNSW-index-size" : 57.676497283391654,
    "LUCENE_HNSW-indexing-time" : 665038,
    "LUCENE_HNSW-mean-latency" : 40.73143438453713,
    "LUCENE_HNSW-mean-retrieval-latency" : 0.0,
    "LUCENE_HNSW-query-throughput" : 23.57211889776772,
    "LUCENE_HNSW-query-time" : 42423,
    "LUCENE_HNSW-recall-accuracy" : 97.4964394710074,
    "LUCENE_HNSW-segment-count" : 25
  }

Not sure if we need to tune these new ivpq parameters better?

    "cuVSIvfPqIndexParamsAddDataOnBuild" : true,
    "cuVSIvfPqIndexParamsCodebookKind" : "PER_SUBSPACE",
    "cuVSIvfPqIndexParamsConservativeMemoryAllocation" : false,
    "cuVSIvfPqIndexParamsForceRandomRotation" : false,
    "cuVSIvfPqIndexParamsKmeansNIters" : 20,
    "cuVSIvfPqIndexParamsKmeansTrainsetFraction" : 0.5,
    "cuVSIvfPqIndexParamsMaxTrainPointsPerPqCode" : 256,
    "cuVSIvfPqIndexParamsMetric" : "L2Expanded",
    "cuVSIvfPqIndexParamsMetricArg" : 2.0,
    "cuVSIvfPqIndexParamsNLists" : 1024,
    "cuVSIvfPqIndexParamsPqBits" : 8,
    "cuVSIvfPqIndexParamsPqDim" : 0,
    "cuVSIvfPqParamsRefinementRate" : 1,
    "cuVSIvfPqSearchParamsInternalDistanceDtype" : "CUDA_R_32F",
    "cuVSIvfPqSearchParamsLutDtype" : "CUDA_R_32F",
    "cuVSIvfPqSearchParamsNProbes" : 20,
    "cuVSIvfPqSearchParamsPreferredShmemCarveout" : 1.0,

No repository file, test, or entry point is identified. Start by reviewing the linked CPU and GPU logs alongside the reported IVF-PQ parameters, dataset details, and benchmark metrics; done should mean determining whether the results come from configuration or an actionable implementation problem.