Lucene: investigate why CAGRA perf is only 1.5x better than HNSW with 1M 1536d 32fp @ 95% recall

As can be seen in the image below, CAGRA is performing about 1.5x better @ 95% recall than HNSW on a dataset that is 1M 1536d 32fp, which is concerningly low.

CPU log files, results, and plots:
https://drive.google.com/file/d/1RYtMjhBrPknoZbkZjGS9tkFZINvNxK5e/view?usp=sharing

GPU log files, results, and plots:
https://drive.google.com/file/d/1HYXRaXLhfv7HDgD3WjIclYHIL7DX5SuR/view?usp=sharing

CPU compressed results spreadsheet:
cpu_f8fEDt_metrics.csv

GPU compressed results spreadsheet:
gpu_mhgiMN_metrics.csv

These are the configs that I used (note: these grids were first-round grids and turned out to be heavy on the 99% recall)

CPU sweeps.json:

{
  "my_dataset-1m": {
    "dataset": "my_dataset-1m",
    "common-params": {
      "numDocs": 1000000,
      "numQueriesToRun": 10000,
      "numWarmUpQueries": 25,
      "flushFreq": 500000,
      "topK": 10000,
      "efSearch": 150,
      "queryThreads": 1,
      "vectorColName": "vector",
      "createIndexInMemory": false,
      "cleanIndexDirectory": true,
      "saveResultsOnDisk": true,
      "forceMerge": [0],
      "enableTieredMerge": true
    },
    "algorithms": {
      "LUCENE_HNSW": {
        "hnswIndexDirPath": "hnswIndex",
        "hnswMaxConn": [24, 32, 64, 96, 128, 160, 192, 224, 256, 288, 320],    
        "hnswBeamWidth": [32, 64, 96, 128, 160, 192, 224, 256, 288, 320, 352],
        "numIndexThreads": [10]
      }
    }
  }
}

GPU sweeps.json:

{
  "my_dataset-1m": {
    "dataset": "my_dataset-1m",
    "common-params": {
      "numDocs": 1000000,
      "numQueriesToRun": 10000,
      "numWarmUpQueries": 25,
      "flushFreq": 500000, 
      "topK": 10000, 
      "efSearch": 150, 
      "queryThreads": 1,
      "vectorColName": "vector",
      "createIndexInMemory": false,
      "cleanIndexDirectory": true,
      "saveResultsOnDisk": true,
      "forceMerge": [0], 
      "enableTieredMerge": true 
    },
    "algorithms": {
      "CAGRA_HNSW": {
        "cuvsIndexDirPath": "cuvsIndex",
        "cagraGraphDegree": [24, 32, 64, 96, 128, 160, 192], 
        "cagraIntermediateGraphDegree": [24, 32, 64, 96, 128, 160, 192], 
        "cagraHnswLayers": [1], 
        "cuvsWriterThreads": 32,
        "numIndexThreads": [10], 
        "hnswMaxConn": [5], 
        "hnswBeamWidth": [5]
      }
    }
  }
}

========
cc: @cjnolet

Start by comparing the linked CPU and GPU logs, results, plots, and compressed metrics spreadsheets against the CPU and GPU sweeps.json configurations. Check how the reported 95% recall comparison changes across the supplied HNSW and CAGRA parameter grids. Done means identifying the cause of the narrow performance gap and documenting an actionable conclusion.