OpenMP thread placement and affinity

In my testing, on a 4-core two-way hyperthreaded Xeon-W Skylake machine, I've found that the following environment variable settings produce consistently high performance:

OMP_NUM_THREADS 4
OMP_PLACES "{0,1,2,3}"
OMP_PROC_BIND spread

This tells the OpenMP library to allow up to 4 threads, tells the OpenMP library that it can start threads on cores 0-3 (and can't use the hyperthreaded siblings 4-7) and that threads should be spread out over the cores as they're started. I believe that it also implies thread affinity so that threads won't move between cores.

I find that if I don't set these environment variables, the performance is generally worse and can also be much more variable. e.g. on a simple test of matrix multiplication, with OMP_NUM_THREADS=4 the run times varied from 6.23 to 8.94 seconds in four tests. After setting OMP_PROC_BIND and OMP_PLACES, the run times varied from 5.44 to 5.49 seconds in four tests.

Is there any more general advice on how to control thread placement and affinity for the best performance with OpenBLAS? What about systems with more cores and multiple sockets? Could information about this be added to the documentation?

Start by reviewing the OpenMP settings and performance measurements described in the issue, including OMP_NUM_THREADS, OMP_PLACES, and OMP_PROC_BIND. Done means adding clear OpenBLAS documentation covering thread placement and affinity for single-socket systems, larger core counts, and multiple sockets.