Cancelling a `fail_after` results in TooSlowError, even though the timeout didn't expire

I've got some code that uses fail_after and cancels itself, example below.
My assumption was that the context manager just exits without any error. Yet it raises a Cancelled and TooSlowError.
When I don't use fail_after but open_cancel_scope, things work as expected. I can't find anything in the docs that explain the difference. And I can't think of any good reason either.
TooSlowError is especially strange because the timeout wasn't reached.

Version: ce131cd519a4ac3cb936ffb9e34e0a4f7e70bc24 on cpython3.6.6

#!/usr/bin/env python

import trio


async def main_a():
    with trio.CancelScope() as cancel_scope:
        await trio.sleep(1)
        cancel_scope.cancel()
        await trio.sleep(1)

    print('Hello!')


async def main_b():
    with trio.fail_after(10) as cancel_scope:
        await trio.sleep(1)
        cancel_scope.cancel()
        await trio.sleep(1)

    print('Hello!')


trio.run(main_a)
print("---")
trio.run(main_b)

Result is:

Hello!
---
Traceback (most recent call last):
  File "/home/joern/.venv/trio/lib/python3.6/site-packages/trio/_timeouts.py", line 116, in fail_at
    yield scope
  File "./bug", line 19, in main_b
    await trio.sleep(1)
  File "/home/joern/.venv/trio/lib/python3.6/site-packages/trio/_timeouts.py", line 85, in sleep
    await sleep_until(_core.current_time() + seconds)
  File "/home/joern/.venv/trio/lib/python3.6/site-packages/trio/_timeouts.py", line 66, in sleep_until
    await sleep_forever()
  File "/home/joern/.venv/trio/lib/python3.6/site-packages/trio/_timeouts.py", line 51, in sleep_forever
    await _core.wait_task_rescheduled(lambda _: _core.Abort.SUCCEEDED)
  File "/home/joern/.venv/trio/lib/python3.6/site-packages/trio/_core/_traps.py", line 165, in wait_task_rescheduled
    return (await _async_yield(WaitTaskRescheduled(abort_func))).unwrap()
  File "/home/joern/.venv/trio/lib/python3.6/site-packages/outcome/_sync.py", line 107, in unwrap
    raise self.error
  File "/home/joern/.venv/trio/lib/python3.6/site-packages/outcome/_async.py", line 19, in capture
    return Value(sync_fn(*args, **kwargs))
  File "/home/joern/.venv/trio/lib/python3.6/site-packages/trio/_core/_run.py", line 629, in raise_cancel
    raise exc
trio.Cancelled

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./bug", line 26, in <module>
    trio.run(main_b)
  File "/home/joern/.venv/trio/lib/python3.6/site-packages/trio/_core/_run.py", line 1334, in run
    raise runner.main_task_outcome.error
  File "./bug", line 19, in main_b
    await trio.sleep(1)
  File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/joern/.venv/trio/lib/python3.6/site-packages/trio/_timeouts.py", line 118, in fail_at
    raise TooSlowError
trio.TooSlowError

Start with trio/_timeouts.py, especially fail_at and the cancellation handling shown in the traceback, then reproduce the issue with the provided main_b example. Compare its behavior with main_a using CancelScope; done means cancelling before the deadline no longer produces TooSlowError, with the expected behavior covered by a regression test.