Unexpected performance degradation when running experiment

Dear OpenBLAS Team,

I'm currently working to improve Numpy's matmul for the strided case and I ran a large grid search with different BLAS frameworks, see

https://github.com/numpy/numpy/pull/23752#issuecomment-2629521597

Here a repost of the plots:

blas_benchmark_v2.pdf

The plots show the improvement of performance of the respective BLAS framework plus copying over naïve matrix multiplication.

In the case of OpenBLAS, I've actually run two experiments: one where the iteration order over the search space is completely shuffled over all "pixels" of the experiment and one where the "pixel"-order is sequential (as is the actual obvious choice, called noshuffle here). In latter noshuffle case, there is an unexpected performance degradation visible as a red triangle in the top right corner, e.g. for n=20 and batch_size=1. That was the reason I've introduced shuffling in the first place. Other frameworks are not affected by iteration order (graphs not provided, but I can do so on demand).

I wonder whether with the help of these plots this performance artefact can be improved. I can do more benchmarks and plots like that if interested and also provide some code.

Best from Berlin, Michael

Start with the linked NumPy PR comment and attached plots; compare the sequential (noshuffle) and shuffled searches at n=20 and batch_size=1. Reproduce the benchmark, then trace the OpenBLAS path involved and document a confirmed cause and measurable improvement; no source file, test, or runnable code is provided.