General performance improvements discussion

Summary:

At the suggestion of @bob-carpenter, I'm opening this as a discussion thread for how to improve the performance of Stan both generally, and on specific platforms. These improvements can vary between simple compile and run-time tricks the user can perform in certain environments which can be addressed with documentation and basic automation, to deep overhauls of the internals to exploit modern processor architecture. While my experience with the code is minimal currently, it seems Stan is already at a very good baseline for performance, but there is likely room for improvement (such as exploiting vectorization better and improving special function performance, for starters).

Description:

Stan is great, and I want to help make it better. This issue is a place to discuss how to best do so, and provide various benchmarks on common systems I have access to see where performance is lagging. The main machines I have readily available for trials are:
AMD Ryzen 5800X (Ubuntu 21.04, Windows 10)
2X AMD EPYC 7742 64-Core Processor (CentOS 7)
2X Intel(R) Xeon(R) Gold 6128 CPU (CentOS 7)
2X Intel(R) Xeon(R) CPU E5-2680 v4 (CentOS 7)
2X Intel(R) Xeon(R) Gold 6148 CPU (CentOS 7)
Fujitsu A64FX (CentOS 8)
ThunderX2 99xx (CentOS 8)
Intel(R) Core(TM) i9-10885H CPU (Ubuntu 21.04)
Intel(R) Core(TM) i7-6920HQ (Big Sur)
Apple M1 (Big Sur)

Benchmarking tools:

I've been using these programs to do some simple benchmarks.
simple cmdstan benchmark
compare amdlibm to libm, need to extract amd-libm somewhere and add the library to library path as in the example script.

Results

This is currently mostly a more qualitative stub, with more results to come later. I'm only using cmdstan so far.
On machines running CentOS 7, glibc is painfully out of date. From a practical standpoint, this means that libm (log, exp, etc.) is also incredibly out of date (2.17). AMD provides a performance tuned version of libm, which honestly is approximately the speed of current libc implementations (given their permissive license, the code might have been absorbed into glibc). However, it absolutely crushes old libc versions.

./test_libm.sh 10
Model name:            Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz
GNU libc version: 2.17
GCC version 7.4

func	libm.so	libalm.so
log	0.3302	0.0555
exp	0.1533	0.0503
log1p	0.1225	0.0994
cos	0.1308	0.0613
sin	0.1379	0.0570
acos	0.1903	0.2063
asin	0.1731	0.2296
atan	0.2069	0.1938
exp10	0.2888	0.0562
exp2	0.1047	0.0518
log10	0.3369	0.0671
log2	0.0930	0.0605
cosh	0.1750	0.2473
sinh	0.1720	0.2661
tanh	0.1916	0.1154
expm1	0.1063	0.0997

Other notes:

1. The code spends a non-negligible amount of time pushing things to the vari_base* stack. This improves with more modern compilers, but is still probably ~5% of execution time on this example for the x86_64 machines. It could be better or worse, it's kinda painstaking to isolate because the stack is modified everywhere.
2. Any BLAS implementation can be swapped in. This can be useful for conda packages, since I doubt you want to auto-pull and build openblas/blis/mkl/whatever.
3. There are likely techniques for more rapid evaluation of heavy 1-dimensional functions, which can really hurt performance on certain problems. I have one such utility for this, but it's on a limited domain/range. There are still probably very good ways to re-tool it to handle these problems.
4. apply_scalar_unary can probably be vectorized for many functions. Vector instructions are currently under-utilized in my limited profiling, but I will investigate this more

Current suggestions

It might be worth detecting the libc implementation (when appropriate) and suggesting the AMD implementation be loaded. This could also be pulled in directly, given their permissive license. The make/local I'm using in cmdstan contains
LDFLAGS+= -L/mnt/home/rblackwell/tmp/amd-libm/lib -Wl,-rpath,"/mnt/home/rblackwell/tmp/amd-libm/lib" -lalm -lm to enable this feature, but this could be changed to $(PWD) or whatever the make equivalent is.

On the same machine/OS, I got significant performance bumps from using a more modern compiler (gcc10 vs gcc7), and swapping the Eigen backend out for MKL, both of which were relatively trivial. The combined effect of these three things cut execution time down from ~75s to ~45s, and it was all very low hanging fruit. I didn't see any obvious mention of this in documentation. The same modifications on my modern Ryzen system didn't do much of anything though (I didn't try an older compiler, MKL and libalm didn't help noticeably). The code executed in about 28-30s, depending on configuration.

Current Version:

v2.27.0

Start by reproducing the supplied CmdStan benchmarks, then inspect CmdStan's make/local, test_libm.sh, apply_scalar_unary, and the vari_base stack references in the issue. The issue has no bounded task or completion criteria, so a contributor would first need to narrow the discussion into a specific benchmarked change.