Lucene: significant variance observed in CAGRA index build recall values

while running CAGRA vs HNSW tests I noticed that one test that should have been 95+ was sub-95, so ran it three more times -- the outcomes of the next three tests were 98, 88, 99, resulting in a standard deviation just above 5!

In order to provide more complete evidence I am attaching a spread sheet titled gpu_FM_uhx1fo_metrics.csv -- columns A through AR represent the original columns from testrun uhx1fo. Columns AU through AX represent relevant columns from testrun YDdVSN which had the exact same parameters as testrun uxx1fo.

column BB measures the average buildtime for each row across two runs (uhx1fo and YDdVSN); column BC measures the average absolute delta in build times.

column BE measures average recall for each row; column BF represents the absolute delta in recall.

notice that about 30% of runs have significant deltas.

I'm not sure if this is normal/expected or not?

===========

my config (sweeps.json) file was:

{
  "my_dataset-1m": {
    "dataset": "my_dataset-1m",
    "common-params": {
      "numDocs": 1000000,
      "numQueriesToRun": 10000,
      "numWarmUpQueries": 25,
      "flushFreq": 500000,
      "topK": 10000,
      "efSearch": 150,
      "queryThreads": 1,
      "vectorColName": "vector",
      "createIndexInMemory": false,
      "cleanIndexDirectory": true,
      "saveResultsOnDisk": true,
      "forceMerge": [1],
      "enableTieredMerge": true
    },
    "algorithms": {
      "CAGRA_HNSW": {
        "cuvsIndexDirPath": "cuvsIndex",
        "cagraGraphDegree": [24, 32, 64, 96, 128, 160, 192, 224],
        "cagraIntermediateGraphDegree": [24, 32, 64, 96, 128, 160, 192, 224],   
        "cagraHnswLayers": [1],
        "cuvsWriterThreads": 32,
        "numIndexThreads": [10],
        "hnswMaxConn": [5],
        "hnswBeamWidth": [5]
      }
    }
  }
}

===========

metrics file:
gpu_FM_uhx1fo_metrics.csv

@cjnolet

Start by reviewing the attached gpu_FM_uhx1fo_metrics.csv and the sweeps.json configuration, then compare the repeated CAGRA_HNSW runs against the HNSW results. Done means establishing whether the recall and build-time deltas are expected or reproducible and documenting the evidence or a clear diagnosis.