python-trio / python-trio/trio
make trio.BlockingTrioPortal and trio.Lock work together
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.3k
- Forks
- 431
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 6
Description
In this SO question, @Nikratio asks about how to share a lock between Trio and a thread.
There are some advantages to using trio.Lock, and having the thread access it through a BlockingTrioPortal – in particular, it means that the Trio-side code can use cancellation as normal. (It's also maybe cheaper? I haven't measured, but it seems plausible.)
However, if you try this, you'll get bitten in a surprising way: trio.Lock tracks which task acquired it, and wants that task to release it. This is intended to help avoid mistakes and give us the option of implementing deadlock detection later (#182), but here it creates a problem: if you do portal.run(lock.acquire), and then later portal.run_sync(lock.release), then you'll get an error, because the BlockingTrioPortal uses a different backing task for each call, so from the Lock's perspective the task that's trying to release it is some random task unrelated to the one that acquired it.
This is unfortunate, we should fix it somehow. Some options:
-
Make
trio.Lockless picky about who releases it. (Likethreading.Lock, which is totally happy for different threads to callacquireandrelease.) Downsides: this would mean we can never add deadlock detection or otherwise report on lock ownership when debugging; feels kind of weird and error-prone; inconsistent with recursive locks (which inherently have to track ownership). Doesn't solve the problem for other context managers that have the same issue (e.g. nurseries – see also https://github.com/python-trio/trio-asyncio/issues/42 which is about context-switching between trio and asyncio, which has essentially isomorphic concerns) -
Add an API
BlockingTrioPortalto wrap a context manager, so you write something likewith portal.async_with(lock): ..., andBlockingTrioPortal.async_withis clever enough to allocate a single backing task and use it for calling both__aenter__and__aexit__. -
Make it so
BlockingTrioPortalsomehow maintains a single backing task across multiple operations automatically. Managing the task lifetime becomes a bit tricky here: what if multiple threads use the same portal? Do we need to add aBlockingTrioPortal.closeAPI to shut down the backing task(s)? I don't think there's any actual show-stoppers that would prevent killing the backing task(s) from__del__, though of course it's pretty complicated to do. Or we could change the API to make this easier to track, e.g. by having awith portal_factory.open() as portal: ...that you have to do in the thread to get aportalhandle? -
Change the trio<->thread API entirely, using something like @agronholm's
async with in_trio: .../async with in_thread: ...trick. https://github.com/python-trio/trio-asyncio/issues/42 has arguments for why this makes sense for trio/asyncio transitions, #649 is the trio issue for discussing the low-level API we need in the core to make this possible. I guess whether this is a good idea depends a lot on how people are using this... do they really want to switch back and forth between trio and a thread? if so it's great. Do they just want to acquire a lock, and otherwise stay in the thread? If so then writingasync with in_trio: async with lock: async with in_thread: ...would be pretty annoying (and there's some question about how to get back into the same thread, or if that even matters). I guess that could be shortened toasync with in_trio, lock, in_thread: ..., but I don't know if that helps much :-).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with trio/_sync.py around lines 583-584 and the BlockingTrioPortal entry points described in the issue. Compare the proposed approaches for preserving lock ownership across portal calls, then determine which API behavior should be supported. Done means a Trio lock can be acquired and released through the portal without the ownership error, with coverage for the chosen behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100