Reduced throughput in sgx_thread_mutex due to fairness?

For a while now we've noticed drops in throughput and stalls in our enclaves when using `std::mutex`, which is backed by `sgx_thread_mutex` in the trusted library. We recently started to look more deeply into this, and ran some benchmarks comparing the regular glibc NPTL mutex implementation to an untrusted model of `sgx_thread_mutex` (so that overhead specifically related to o-calls is discounted). It turns out that in a basic throughput test, the SGX SDK mutex design consistently achieves only about 6% - 9% of the rate of lock acquires per second as the NPTL mutex, across all contention levels.

There seem to be several factors contributing to this (especially in higher contention levels), but one of the main ones appears to be the strictly fair design `sgx_thread_mutex`. A thread is only allowed to acquire the mutex if there are no pending threads, or if it is itself the next in line for the mutex. This means that a thread which released a mutex and is then intreseted in re-acquiring it (which is a common enough flow) is be forced to leave the enclave and sleep on its' futex, while in the meanwhile the critical section is left empty as the next thread in line wasn't yet even scheduled to run by the kernel.

Designing general purpose locks to be strictly fair lock designs seem to be the exception rather than the norm, so I wonder why did the designers of `sgx_thread_mutex` chose to do it like this, rather than allowing active threads to barge into the mutex. Are there any safety or security benefits here that I may have overlooked?

Thanks!

Start with the trusted-library implementation of `sgx_thread_mutex` and trace its interaction with the futex path described in the issue. Compare the fairness behavior against the cited glibc NPTL mutex model and the throughput benchmarks; the work is complete when the fairness rationale, security implications, and any justified design change are established.

Written by the indexing model from the issue text.