For sane uniform error handling we need an MPI Barrier with timeout.

This can be accomplished using MPI_ibarrier. It's probably not too hard to add support for this to mpi3 although it's clearly complicated by mpi3's communication modes. Alfredo I'm not really grasping how the sync and async MPI calls and the various apparently sync and async communication modes interact.

Our use case is after a fatal error has occurred which is likely to occur on all nodes and therefore frequently preempt reporting by the head node. So we could just spin on an MPI_probe for a specified timeout the process that would be wasting CPU spinning is already "dead"

Still we only need an ibarrier call that returns a request and a probe command and we can implement the timeout and spin at the application level.