Optimizing ibverbs network performance

Nobody has claimed this yet.

Eventually, we'll want to be able to switch from TCP to ibverbs to pick up better performance.

To communicate with ibverbs, one has to register memory with the network card, and that registration step has high overhead. To work around this, it's common to pre-register memory regions with the card and then use those regions for network buffers. It looks like margo provides some help with that by using their bulk pools:

https://xgitlab.cels.anl.gov/sds/margo/-/blob/master/include/margo-bulk-pool.h

Most MPI implementations on verbs use pre-pinned memory buffers to transfer small and medium data. They will copy data from the original user buffer into these buffers and send from there (memcpy cost is lower than registration). For large transfers (e.g., MB range), they will dynamically pin the user buffer and send from it, since at that point (memcpy cost is higher than registration).

This refactoring may push us to packetize messages that we are not packetizing right now.

Start by tracing the current TCP data path and reviewing the linked Margo bulk-pool reference. Compare the existing message handling with the proposed pre-registered buffers, dynamic registration, and packetization approach. Done would require a defined implementation plan and verified ibverbs performance behavior, but the issue names no files or tests.

Written by the indexing model from the issue text.