Eliminate HAVE_MPI conditional compilation

We made the MPI code conditionally compiled originally on the idea that maybe multiple forms of parallelism would be supported here. But, do we expect to be doing anything useful at large scale without MPI? If not, having MPI conditionally compiled only complicates the code without much benefit and it should be removed.

Search the repository for HAVE_MPI conditional-compilation guards and read the surrounding MPI implementation and build configuration. Confirm the full scope of the guards, then run the repository's existing build or test commands; done means the conditional MPI paths are removed without breaking the MPI-enabled build.

Written by the indexing model from the issue text.