RFC: Interruption model

Nobody has claimed this yet.

The Linux shim needs to be able to cause a remote thread to interrupt its current work. This is needed for signals (since the target thread may be indefinitely running in the guest or waiting in a blocking syscall) and for multi-threaded process termination (since the shim needs to get all the process's threads to a termination point). Without this capability, the remote threads may never check for this remotely initiated work, which will manifest as hangs.

# Thread states

There are three different states to think about for interruption:
1. The remote thread is running in the guest.
2. The remote thread is running in the platform or shim.
3. The remote thread is blocked on some litebox wait API.

## Running in guest

Interrupting a thread running in the guest requires new platform support. The most reasonable extension of the litebox platform traits would be to add an interrupt method to the `ThreadProvider`:

```rust
fn interrupt_thread(&self, thread: &Self::Thread);
```

In this case, `Thread` is a new associated type that is returned when a thread is created (the shim will also provide some way to get this value for the initial thread). This function would cause the platform to interrupt the thread if it's running in the guest, save the context, then call back into the shim via some kind of shim API:

```rust
fn handle_interruption(ctx: &mut PtRegs) -> ContinueOperation
```

To simplify platform implementations, we can say that a platform is allowed to call this function at any time; the shim is responsible for handling spurious interruptions.

The Linux userland platform would implement this mechanism by sending a signal to the target thread. The signal handler, only if it detected that the thread is currently running in the guest, would save the current context and then update the RIP to jump back to the platform asm code. This code would call into the shim to handle the interruption. Windows userland would have a similar implementation based on `SuspendThread`+`GetThreadContext`+`SetThreadContext`.

Kernel-based platforms would likely send an IPI to cause the target CPU to take an interrupt in kernel mode. The interrupt handler would save the context and run the platform code for jumping back into the shim.

## Running in platform or shim

In the case where the remote thread is running in the platform or shim and is not blocked in a litebox wait API, the `interrupt_thread` call should pend the interrupt request. When the platform is about to return to user mode, it should check for this pended state and call `handle_interruption` immediately instead of running the guest.

This can't be handled generically by the shim or litebox, because only the platform code can resolve the race condition between starting to return to the guest and the interruption arriving. Different platforms will manage this in different ways. E.g., the Linux userland platform's signal handler will need to look at the RIP--if it's in the window after the interruption state has been checked but before the context has been fully restored, then it will need to abort the context restore and resume back into the shim. Windows userland will need to handle this slightly differently since there are two different context restore paths, one of which call into a system DLL.

## Blocked in wait API

The most interesting case from a design perspective is when the thread is blocked in a litebox wait API. In this case, the Linux shim wants some litebox waits to be interruptible, so that an interrupt causes them to return with a special error code. This would be used for waits in syscalls such as `poll`, `nanosleep`, and `futex`, which should all return `EINTR` when there's a pending signal to handle (modulo `sigaction`'s opt-in `SA_RESTART` support). But not all waits should be interruptible--for example, internal shim mutexes should generally use uninterruptible waits, since those are not expected to be blocked for extended periods, and it's not desirable to require all mutex acquires to handle the interruption failure path.

To support this, litebox needs to add support for interruptible waits.

There are two ways we can achieve this requirement:
1. Handle this within the shim/litebox without platform involvement. The shim can do this when requesting an interrupt by checking if the remote thread is known to be an interruptible wait and, if so, waking it via the `RawMutex` mechanisms instead of (or in additional to) the `interrupt_thread` mechanism. These contexts are mostly already using an abstraction (such as `litebox::event::Pollee`) that could be extended with this support. We could add this concept to litebox itself.
2. Extend the platform `RawMutexProvider` to support interruptions. The semantics would be that if `block` is called with `wake_on_interrupt = true`, then the call to `block` returns as soon as there is a pending interruption for the thread.

Each approach has tradeoffs.

### Litebox implementation

This approach can be implemented without platform changes, by having litebox track thread state and use the right interruption mechanism depending on that state.

In this model, Litebox would probably use a single per-thread `RawMutex` value for all interruptible waits. When the shim wants to interrupt a wait, it calls into litebox to wake the per-thread raw mutex with a special `INTERRUPTED` value.

A possible downside of this approach is that `SIGALRM` handling is somewhat difficult to implement efficiently. For `SIGALRM`, the shim needs a mechanism to register for an interruption after a certain amount of time has passed. This requires platform support. Litebox will need to wrap this platform functionality as well, to make sure that waits are interrupted by the timeout (presumably by overriding the timeout values of the waits with the next alarm time). This will probably cause duplicate timer programming in the platform code.

### Platform implementation

In this approach, the platform manages thread state and implements interruptions correctly and efficiently for the different cases. When `interrupt_thread` is called, the platform ensures that a pending interruption is registered. If the thread is currently in an interruptible wait, then the wait returns with an error code indicating there's a pending interruption.

This is nice because it gives litebox and the shim more flexibility to use the `RawMutex` API in clever ways while supporting interruptible waits. With the other approach, all interruptible waits must be coordinated through litebox so that the target `RawMutex` can be identified and woken.

However, the downside of this approach is that it is difficult for platforms to implement:
* Linux userland uses `futex` for `RawMutex` wakes. `futex` does not provide a way to atomically change the signal mask as part of a futex wait, so although it returns `EINTR` if a signal is delivered, it does not get cancelled if the signal is delivered _just before_ the call to `futex`. Working around this is complicated and hard to get right--the signal handler would probably need to look at the instruction pointer and rewind to just before the pending interruption check to mitigate this.
* Windows uses `WaitOnAddress`. This provides no mechanism for interruptions (e.g., via APCs). We would need to rewrite this to use the lower-level `NtAlertThreadByThreadId`-type calls to have finer grained control.

And this would put similarly annoying constraints on any new platforms that come later.

Given these challenges, I think we should implement this in litebox itself and not in the individual platforms.

This is an RFC rather than a file-specific task; start by reviewing the ThreadProvider, RawMutexProvider, litebox event::Pollee, and shim interruption concepts described here. The work is done when a settled litebox-based interruption model covers guest execution, platform or shim execution, and interruptible waits, with the required platform interactions defined.

Written by the indexing model from the issue text.