fsspec / fsspec/filesystem_spec
error using fsspec and flask-socketio
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
I am trying to incorporate calls to fsppec-based packages (adlfs, s3fs) into a (sync) flask view.
Flask is running using flask-socketio, which is in turn using gevent.
Whenever I try and call a filesystem method (e.g. ls), it raise the following error
NotImplementedError: Calling sync() from within a running loop
I can bypass the issue by telling flask-socketio to use threading and not gevent (and remove the patching), however this has limitations (i.e. it uses werkzeug server, which is dev only) which means it can't be used in production.
minimal working example:
server.py
from gevent import monkey
monkey.patch_all()
import os
from adlfs import AzureBlobFileSystem
from azure.identity.aio import DefaultAzureCredential
from flask import Flask
from flask_socketio import SocketIO
def create_app():
app = Flask(__name__)
@app.route("/")
def hello_world():
return "<p>Hello, World!</p>"
@app.route("/ls")
def listing():
credentials = DefaultAzureCredential()
account_name = os.environ.get("AZURE_ACCOUNT_NAME")
azure_path = os.environ.get("AZURE_PATH")
new_fs = AzureBlobFileSystem(account_name=account_name, credential=credentials)
res = new_fs.ls(azure_path)
list_str = [f"<p>{l}</p>" for l in res]
return f"<p>Hello, Listing!</p>{''.join(list_str)}"
return app
python server.py
curl 127.0.0.1:5000/ls
results in
[2024-10-02 16:57:35,406] ERROR in app: Exception on /ls [GET]
Traceback (most recent call last):
File "<path>/site-packages/flask/app.py", line 2073, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<path>/site-packages/flask/app.py", line 1519, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<path>/site-packages/flask/app.py", line 1517, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "<path>/site-packages/flask/app.py", line 1503, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "server.py", line 26, in listing
res = new_fs.ls(azure_path)
^^^^^^^^^^^^^^^^^^^^^
File "<path>/site-packages/fsspec/asyn.py", line 118, in wrapper
return sync(self.loop, func, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<patch>/site-packages/fsspec/asyn.py", line 80, in sync
raise NotImplementedError("Calling sync() from within a running loop")
NotImplementedError: Calling sync() from within a running loop
requirements.txt
adlfs==2024.7.0
Flask==2.1.3
Flask-SocketIO==5.3.6
fsspec==2024.6.1
gevent==24.2.1
azure-identity==1.17.1
werkzeug=2.2.2
Is there a known workaround for this? I have tried various alternatives, to no avail.
e.g.
- passing a loop to the filesystem init, then calling
_ls - calling
asyncio.run(fs.ls)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the minimal example in server.py, especially the listing view and its AzureBlobFileSystem.ls call, then reproduce it with the versions in requirements.txt. Read the fsspec/asyn.py sync traceback and compare the gevent and threading configurations. Done means identifying and documenting a supported workaround or a narrowly scoped fix for synchronous filesystem calls inside this setup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, flask, python
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100