fsspec / fsspec/s3fs

Runtime Error: Loop is not running

Open
#768 13 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Description

Versions:

fsspec                        2023.6.0
s3fs                          2023.6.0

I'm trying to use a basic method .get, inside of a fast API route. Like this:

@app.post("/run_job")
async def run_job(payload: JobPayload):
    metadata = payload.metadata
    job_id, job_type, creds = metadata.job_id, metadata.job_type, metadata.s3_creds

    job_dir = get_job_dir(job_id)
    job_dir.mkdir()

    import asyncio

    s3 = S3FileSystem(
        key=creds.access_key,
        secret=creds.secret_key,
        client_kwargs={"endpoint_url": creds.endpoint, "region_name": creds.region},
        asynchronous=True,
        loop=asyncio.get_event_loop(),
    )
    session = await s3.set_session()
    await s3._get(some_value.rstrip('/') + "/", local_path, recursive=True)

    await session.close()

This does not work, but using aiobotocore directly does. What am I doing wrong?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the shown FastAPI route with fsspec 2023.6.0 and s3fs 2023.6.0, then inspect the S3FileSystem asynchronous setup, set_session, and _get entry points. Done means identifying why the event loop is not running and documenting or validating a corrective change.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, fastapi, python
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.