fsspec / fsspec/s3fs

FastAPI async endpoint: which s3fs client should I use? Needed example

Open
#907 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Description

If I wrap the s3fs client inside a DAO class in order to create an interface to add features with the remote fs, which s3fs client should I use?

For instance, assuming I create an async client, because I wish to use either example_one or example_two "coroutines/methods" in FastAPI async endpoints, which example should I use?

class S3Manager:
    def __init__(self, bucket_name: str = None):
        self.bucket_name = bucket_name
        self.s3_fs = s3fs.S3FileSystem(
            anon=False,
            asynchronous=True,
        )
        
    async def example_one(self):
        session = await self.s3_fs.set_session()
        work = await self.s3_fs._glob(...)
        await session.close()
        return work
        
    def example_two(self):
        return s3_fs.glob(...)

Since FastAPI handles its own loop to manage async coroutines, I believe the best approach would be to create the async client, leveraging the same FastAPI loop somehow: e.g. delegating the management to it. But:

  1. the async client and example seems cumbersome: it seems like the s3fs library wishes to user to exploit the sync interface/client which itself uses the async support behind.
  2. is there a performance gap by using the sync blocking client by calling its methods in a async coroutine (fastapi)?
  3. how should I setup a complete example using FastAPI own loop?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the issue's S3Manager examples and the s3fs.S3FileSystem asynchronous=True setup, then compare the sync and async methods in a FastAPI async endpoint. Document which client pattern should be used, whether blocking calls affect the event loop, and provide a complete example if the project supports one.

Written by the indexing model from the issue text.

Assessment

Tech stack
fastapi, python
Domain
api, backend, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.