crewAIInc / crewAIInc/crewAI

UploadCache sync wrappers deadlock when called from inside a running event loop

Open
#6,742 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

no-issue-activity
Dominant language
Python
Stars
58.8k
Forks
8.5k
Avg merge
1d 15h
Merged PRs (30d)
109

Description

Description

UploadCache._run_sync has a branch for being called while an event loop is already running. Its docstring says it runs the coroutine "without blocking event loop". It does the opposite: it schedules the coroutine on the caller's own running loop and then blocks the caller's thread waiting for it. That thread is the thread running the loop, so the loop can never advance the coroutine it was just handed. The call stalls for the full 30-second timeout and raises TimeoutError.

lib/crewai-files/src/crewai_files/cache/upload_cache.py:402-413 on main (ebe0082):

@staticmethod
def _run_sync(coro: Any) -> Any:
    """Run an async coroutine from sync context without blocking event loop."""
    try:
        loop = asyncio.get_running_loop()
    except RuntimeError:
        loop = None

    if loop is not None and loop.is_running():
        future = asyncio.run_coroutine_threadsafe(coro, loop)
        return future.result(timeout=30)
    return asyncio.run(coro)

asyncio.run_coroutine_threadsafe is documented as being for submitting a coroutine to a loop running in a different thread. Here loop was obtained from asyncio.get_running_loop(), i.e. it is this thread's loop, so the precondition is inverted.

Scope

Every one of the ten synchronous wrappers funnels through _run_sync, so all of them hang when called from async code:

wrapper line
get :417
get_by_hash :424
set :438
set_by_hash :453
remove :462
remove_by_file_id :467
clear_expired :472
clear :477
get_all_for_provider :482

UploadCache and cleanup_expired_files are public exports of crewai_files, so this is reachable from user code: a FastAPI handler, a notebook cell, or an async crew callback that touches the upload cache synchronously will stall for 30s and then raise.

Steps to reproduce

import asyncio

from crewai_files import FileBytes, ImageFile
from crewai_files.cache.upload_cache import UploadCache

MINIMAL_PNG = (
    b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01"
    b"\x08\x06\x00\x00\x00\x1f\x15\xc4\x89\x00\x00\x00\nIDATx\x9cc\x00"
    b"\x01\x00\x00\x05\x00\x01\r\n-\xb4\x00\x00\x00\x00IEND\xaeB`\x82"
)


def use_the_cache() -> None:
    cache = UploadCache()
    file = ImageFile(source=FileBytes(data=MINIMAL_PNG, filename="test.png"))
    cache.set(file=file, provider="gemini", file_id="file-123")
    print("cached:", cache.get(file=file, provider="gemini"))


use_the_cache()  # no running loop -> works


async def main() -> None:
    use_the_cache()  # a loop is running -> hangs 30s, then TimeoutError


asyncio.run(main())

Output:

cached: CachedUpload(file_id='file-123', ...)
Traceback (most recent call last):
  ...
  File ".../crewai_files/cache/upload_cache.py", line 412, in _run_sync
    return future.result(timeout=30)
  File ".../concurrent/futures/_base.py", line 458, in result
    raise TimeoutError()
TimeoutError

The bridging logic alone reproduces it, with no cache involved:

import asyncio


async def work():
    return "ok"


def run_sync(coro):
    loop = asyncio.get_running_loop()
    return asyncio.run_coroutine_threadsafe(coro, loop).result(timeout=3)


async def main():
    try:
        return run_sync(work())
    except BaseException as e:
        return f"{type(e).__name__} (deadlock)"


print("inside a running loop:", asyncio.run(main()))
# -> inside a running loop: TimeoutError (deadlock)

Expected behavior

The synchronous wrappers return their result whether or not the calling thread has a running event loop. Providing a sync surface over the async cache is the entire purpose of these wrappers, and the loop.is_running() branch exists specifically to handle this case.

Suggested fix

Drive the coroutine on a worker thread with its own loop, so the thread that blocks is not the thread that must make progress:

if loop is not None and loop.is_running():
    executor = ThreadPoolExecutor(max_workers=1)
    try:
        return executor.submit(asyncio.run, coro).result(timeout=30)
    finally:
        executor.shutdown(wait=False)

wait=False matters: the context-manager form's shutdown(wait=True) blocks until the worker finishes, which would make the 30s bound meaningless on exactly the timeout it is supposed to guard.

Loop affinity is not a concern here — the pre-existing else branch already ran every coroutine under a fresh asyncio.run, so the aiocache backend was already required to be loop-agnostic, and the default memory backend is a plain dict.

lib/crewai-files/tests/test_upload_cache.py currently has no async coverage at all, which is why a branch that always timed out went unnoticed.

Additional context

This is the deadlock half of the same sync/async bridging class as camel-ai/camel#4239: a run_coroutine_threadsafe call whose target loop is the caller's own. I have a patch with regression tests ready and will open a PR referencing this issue.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read lib/crewai-files/src/crewai_files/cache/upload_cache.py around _run_sync and the synchronous wrappers, then reproduce the failure from the issue. Add async regression coverage in lib/crewai-files/tests/test_upload_cache.py and verify that each sync wrapper returns under a running event loop without the timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.