Select random target for drmemtrace scheduler direct switch failures

Some direct thread switch uses (TRACE_MARKER_TYPE_DIRECT_THREAD_SWITCH) are really just yields in a user-mode scheduler. That scheduler picks whoever is next available; there is no data dependence or targeted handoff.

This means that during replay, if the schedule/timing is a little different from the as-traced run, a direct switch target may already be running, since it wasn't just waiting but was ready in the user mode scheduler runqueue. We see this manifest as a significant fraction of the direct switch targets failing. This by itself doesn't really affect representativeness. However, this ends up reducing the migration count, which does affect performance.

This is a feature proposal to pick a new target randomly if the recorded direct switch target is unavailable. The target should be picked only from a set of threads previously seen as direct switch targets.

A possible extension is to turn some waits (TRACE_MARKER_TYPE_SYSCALL_UNSCHEDULE) into direcrt switches. We observe a higher fraction of waits in traces than the original workload. The theory is the trace overhead causes many threads to be blocked at trace i/o and thus on a yield there is more likely to be no one to run, in which case the user mode scheduler emits wait instead of switch.