[discussion] What to do about cancel scopes / nurseries inside generators?

So here's a surprising thing that can happen in trio right now: if you have a generator (or async generator, doesn't matter) that yields inside a cancel scope, then the cancel scope remains in effect until you resume the generator and exit it.

For context managers, this is actually what you want, and we make use of this in lots of places, e.g.:

@contextmanager
def fail_at(deadline):
    with move_on_at(deadline) as scope:
        yield scope
    ...

with fail_at(deadline):
    XX

So here we have: first fail_at runs for a bit, and it starts a cancel scope, and then it yields. And then the XX code runs with the cancel scope in effect, even though failt_at is not on the stack. And then fail_at is resumed, and it exits the cancel scope. A little confusing when you think about the details, but basically this is safe and useful and ultimately does what you'd expect.

However, in general this can cause strange behavior, and also crashes:

import trio

def weird_generator():
    with trio.open_cancel_scope() as cs:
        yield cs

async def main():
    # hold onto a reference to prevent the gc from making this even more confusing:
    gen = weird_generator()
    for cs in gen:
        break
    cs.cancel()
    await trio.sleep(1)

trio.run(main)

Notice that even after we leave the loop and abandon the generator, the cancel scope is still in effect... and then the program crashes because trio detects that the cancel scope stack has become inconsistent (it tries to pop a scope that isn't at the top of the stack).

This all also applies to using nurseries inside async generators, because nurseries have an implicit cancel scope.

There are also some more generic issues with cleaning up (async) generators that call trio APIs after yielding, but that's another issue (specifically, issue #265). The cancel scope leakage causes problems even while the generator is alive, and even if we had PEP 533 then this would still be an issue.

Really the ideal solution here would be to error out if someone tries to yield through a cancel scope, IFF they're not implementing a context manager.

I don't think there's any way to do this though without PEP 521, and even then we'd have to figure out somehow whether this particular generator is implementing a context manager. (I guess some annoying thread-local thing could work -- e.g. make with trio.this_is_a_context_manager: ... which sets some internal flag that disables the PEP 521-based checks.)

I don't currently have any other ideas how to improve this, except to put some prominent warning in the documentation.

Start with the generator example using trio.open_cancel_scope, trio.sleep, and trio.run, then read the discussion alongside issue #265 and the linked PEP 533 and PEP 521. The issue presents several possible approaches but does not select one or define a testable completion condition.