UploadCache sync wrappers deadlock when called from inside a running event loop
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 58.8k
- Forks
- 8.5k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 109
Description
Description
UploadCache._run_sync has a branch for being called while an event loop is already running. Its docstring says it runs the coroutine "without blocking event loop". It does the opposite: it schedules the coroutine on the caller's own running loop and then blocks the caller's thread waiting for it. That thread is the thread running the loop, so the loop can never advance the coroutine it was just handed. The call stalls for the full 30-second timeout and raises TimeoutError.
lib/crewai-files/src/crewai_files/cache/upload_cache.py:402-413 on main (ebe0082):
@staticmethod
def _run_sync(coro: Any) -> Any:
"""Run an async coroutine from sync context without blocking event loop."""
try:
loop = asyncio.get_running_loop()
except RuntimeError:
loop = None
if loop is not None and loop.is_running():
future = asyncio.run_coroutine_threadsafe(coro, loop)
return future.result(timeout=30)
return asyncio.run(coro)
asyncio.run_coroutine_threadsafe is documented as being for submitting a coroutine to a loop running in a different thread. Here loop was obtained from asyncio.get_running_loop(), i.e. it is this thread's loop, so the precondition is inverted.
Scope
Every one of the ten synchronous wrappers funnels through _run_sync, so all of them hang when called from async code:
| wrapper | line |
|---|---|
get |
:417 |
get_by_hash |
:424 |
set |
:438 |
set_by_hash |
:453 |
remove |
:462 |
remove_by_file_id |
:467 |
clear_expired |
:472 |
clear |
:477 |
get_all_for_provider |
:482 |
UploadCache and cleanup_expired_files are public exports of crewai_files, so this is reachable from user code: a FastAPI handler, a notebook cell, or an async crew callback that touches the upload cache synchronously will stall for 30s and then raise.
Steps to reproduce
import asyncio
from crewai_files import FileBytes, ImageFile
from crewai_files.cache.upload_cache import UploadCache
MINIMAL_PNG = (
b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01"
b"\x08\x06\x00\x00\x00\x1f\x15\xc4\x89\x00\x00\x00\nIDATx\x9cc\x00"
b"\x01\x00\x00\x05\x00\x01\r\n-\xb4\x00\x00\x00\x00IEND\xaeB`\x82"
)
def use_the_cache() -> None:
cache = UploadCache()
file = ImageFile(source=FileBytes(data=MINIMAL_PNG, filename="test.png"))
cache.set(file=file, provider="gemini", file_id="file-123")
print("cached:", cache.get(file=file, provider="gemini"))
use_the_cache() # no running loop -> works
async def main() -> None:
use_the_cache() # a loop is running -> hangs 30s, then TimeoutError
asyncio.run(main())
Output:
cached: CachedUpload(file_id='file-123', ...)
Traceback (most recent call last):
...
File ".../crewai_files/cache/upload_cache.py", line 412, in _run_sync
return future.result(timeout=30)
File ".../concurrent/futures/_base.py", line 458, in result
raise TimeoutError()
TimeoutError
The bridging logic alone reproduces it, with no cache involved:
import asyncio
async def work():
return "ok"
def run_sync(coro):
loop = asyncio.get_running_loop()
return asyncio.run_coroutine_threadsafe(coro, loop).result(timeout=3)
async def main():
try:
return run_sync(work())
except BaseException as e:
return f"{type(e).__name__} (deadlock)"
print("inside a running loop:", asyncio.run(main()))
# -> inside a running loop: TimeoutError (deadlock)
Expected behavior
The synchronous wrappers return their result whether or not the calling thread has a running event loop. Providing a sync surface over the async cache is the entire purpose of these wrappers, and the loop.is_running() branch exists specifically to handle this case.
Suggested fix
Drive the coroutine on a worker thread with its own loop, so the thread that blocks is not the thread that must make progress:
if loop is not None and loop.is_running():
executor = ThreadPoolExecutor(max_workers=1)
try:
return executor.submit(asyncio.run, coro).result(timeout=30)
finally:
executor.shutdown(wait=False)
wait=False matters: the context-manager form's shutdown(wait=True) blocks until the worker finishes, which would make the 30s bound meaningless on exactly the timeout it is supposed to guard.
Loop affinity is not a concern here — the pre-existing else branch already ran every coroutine under a fresh asyncio.run, so the aiocache backend was already required to be loop-agnostic, and the default memory backend is a plain dict.
lib/crewai-files/tests/test_upload_cache.py currently has no async coverage at all, which is why a branch that always timed out went unnoticed.
Additional context
This is the deadlock half of the same sync/async bridging class as camel-ai/camel#4239: a run_coroutine_threadsafe call whose target loop is the caller's own. I have a patch with regression tests ready and will open a PR referencing this issue.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read lib/crewai-files/src/crewai_files/cache/upload_cache.py around _run_sync and the synchronous wrappers, then reproduce the failure from the issue. Add async regression coverage in lib/crewai-files/tests/test_upload_cache.py and verify that each sync wrapper returns under a running event loop without the timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 30/100