A primitive for suspending the event loop

As you'd expect for a structured concurrency system, Trio's "event loop" (trio.run()) always runs from start to finish -- you can't stop it partway through, do something else, and restart it again. This is definitely a good default, but there are some cases where I think a "suspend" primitive would be useful. I figured I would create an issue to figure out whether it's a good idea before doing a PR.

The proposed primitive

Inside a Trio task, you can say await trio.hazmat.suspend_trio_from_async(sync_fn, *args). This will:

• leave "Trio context" (stashing and clearing the trio._core._run.GLOBAL_RUN_CONTEXT thread-locals runner and task)
• leave the calling task's contextvars context
• run sync_fn(*args)
• reenter "Trio context" (restoring the stashed thread-locals) and the calling task's contextvars context
• return or raise (back into the task that called suspend_trio_from_async) whatever sync_fn(*args) returned or raised

In a non-task Trio context such as an abort_fn or IOManager implementation, you can say trio.hazmat.suspend_trio_from_sync(sync_fn, *args). This works like suspend_trio_from_async except that it only needs to save and restore runner.

You can implement one in terms of the other, by taking advantage of the fact that abort_fns execute in a non-task context:

async def suspend_trio_from_async(sync_fn, *args):
    result = None
    def abort_fn(_):
        nonlocal result
        result = outcome.capture(suspend_trio_from_sync, sync_fn, *args)
        return trio.hazmat.Abort.SUCCEEDED
    with trio.CancelScope(shield=True) as scope:
        scope.cancel()
        await trio.hazmat.wait_task_rescheduled(abort_fn)
    return result.unwrap()

def suspend_trio_from_sync(sync_fn, *args):
    from trio._core._run import GLOBAL_RUN_CONTEXT as ctx
    assert hasattr(ctx, "runner")
    assert not hasattr(ctx, "task")
    saved_runner = ctx.runner
    del ctx.runner
    try:
        return sync_fn(*args)
    finally:
        ctx.runner = saved_runner

Some ways to use it

• If you want to nest a call to trio.run() inside trio.run(), you can pass trio.run as the sync_fn. This isn't usually a good idea (since the outer run() will block waiting for the inner one) but for a short-lived inner task it might make sense, and be required for encapsulation reasons.

• If you're implementing a foreign event loop API without necessarily-start-to-finish run() semantics on top of Trio (as in trio_asyncio.SyncTrioEventLoop), you can use greenlet to switch away from the Trio run stack to a different stack, implementing that event loop's stop() as something that bottoms out in a suspend call with sync_fn = some_greenlet.switch.

• If we add pluggable IOManagers as has been suggested for #399 (qt support), and you want to support Trio+(other-opinionated-event-loop) interop, you could have an IOManager whose handle_io() call performs suspend_trio_from_sync(some_greenlet.switch) instead of sleeping. Then the other event loop would switch back to the Trio stack when Trio's nearest deadline expires or some I/O happens on an fd in which Trio is interested. (It's unclear to me whether this will work better or worse in the long run than the threaded approach suggested toward the end of #399, but it seems useful to have the necessary machinery to support both.)

• If you want to fork off a child that's not confused by appearing to still be in Trio context, you could suspend with the sync_fn as something like

def fork_child():
    if (pid := os.fork()) != 0:
        return pid
    try:
        run_child()
    finally:
        os._exit(0)

• It's possible to use greenlets plus this suspend primitive to make syntactically-synchronous expressions perform async operations within the same task. Details are left as an exercise to the reader.

Possible problems

If you use this to run multiple calls to trio.run() in the main thread, they'll fight over the global signal state (wakeup_fd, KI handler, any other signal handlers). Can probably be resolved by saving/restoring the global signal state alongside GLOBAL_RUN_CONTEXT.runner.

It's weird. The things it lets you do are really weird. On the other hand, the core surface area of the change is small, and if it makes interop with other event loops easier and more robust, it might still be a net reduction in complexity for the ecosystem as a whole.

Start by reading trio.run(), trio.hazmat.wait_task_rescheduled(), and trio._core._run.GLOBAL_RUN_CONTEXT, then review the proposed suspend_trio_from_async and suspend_trio_from_sync behavior. Check the related interop context in #399 and determine how signal state and context restoration should work. Done means an agreed design and implementation with coverage for the stated suspension and resumption cases.