sgemv() not scaling well on AMD ZEN 2 / EPYC 7742

Hi,

I benchmarked OpenBLAS' 0.3.8 sgemv() on a dual AMD EPYC 7742 (64 cores each) for a 4000x4000 matrix and I find that the performance maxes out at about 48 threads. The peak in the histogram of 1,000,000 repetitions in a for-loop is at around 43 µs. Using 64 threads, the performance drops to about 48 µs, and to around 70 µs (with a lot of jitter to about 160 µs) if I use 128 threads. In the case of 48 or 64 threads, I limit the threads to one CPU using taskset.

If I run 4 separate processes simultaneously with 32 threads each (each process is limited to an exclusive set of 32 cores) doing a 1000x4000 MVM, the timing for each MVM gets down to around 22 µs, which is about what I would expect from the scaling I see up to 48 cores. (If only one 32 thread process is running doing the 1000x4000 MVM, the timing is around 18 µs).

Start by reproducing the reported sgemv() benchmarks from the issue on a dual AMD EPYC 7742 system, comparing one process with 32, 48, 64, and 128 threads against four 32-thread processes. The issue names no source file or test; done would require identifying and correcting the scaling regression, or documenting its cause and verified limits.