fsspec / fsspec/filesystem_spec

error using fsspec and flask-socketio

Open
#1,701 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

I am trying to incorporate calls to fsppec-based packages (adlfs, s3fs) into a (sync) flask view.
Flask is running using flask-socketio, which is in turn using gevent.
Whenever I try and call a filesystem method (e.g. ls), it raise the following error
NotImplementedError: Calling sync() from within a running loop

I can bypass the issue by telling flask-socketio to use threading and not gevent (and remove the patching), however this has limitations (i.e. it uses werkzeug server, which is dev only) which means it can't be used in production.

minimal working example:

server.py

from gevent import monkey

monkey.patch_all()

import os

from adlfs import AzureBlobFileSystem
from azure.identity.aio import DefaultAzureCredential
from flask import Flask
from flask_socketio import SocketIO


def create_app():
    app = Flask(__name__)

    @app.route("/")
    def hello_world():
        return "<p>Hello, World!</p>"

    @app.route("/ls")
    def listing():
        credentials = DefaultAzureCredential()
        account_name = os.environ.get("AZURE_ACCOUNT_NAME")
        azure_path = os.environ.get("AZURE_PATH")
        new_fs = AzureBlobFileSystem(account_name=account_name, credential=credentials)
        res = new_fs.ls(azure_path)
        list_str = [f"<p>{l}</p>" for l in res]
        return f"<p>Hello, Listing!</p>{''.join(list_str)}"

    return app
python server.py
curl 127.0.0.1:5000/ls

results in

[2024-10-02 16:57:35,406] ERROR in app: Exception on /ls [GET]
Traceback (most recent call last):
  File "<path>/site-packages/flask/app.py", line 2073, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<path>/site-packages/flask/app.py", line 1519, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<path>/site-packages/flask/app.py", line 1517, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "<path>/site-packages/flask/app.py", line 1503, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "server.py", line 26, in listing
    res = new_fs.ls(azure_path)
          ^^^^^^^^^^^^^^^^^^^^^
  File "<path>/site-packages/fsspec/asyn.py", line 118, in wrapper
    return sync(self.loop, func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<patch>/site-packages/fsspec/asyn.py", line 80, in sync
    raise NotImplementedError("Calling sync() from within a running loop")
NotImplementedError: Calling sync() from within a running loop

requirements.txt

adlfs==2024.7.0
Flask==2.1.3
Flask-SocketIO==5.3.6
fsspec==2024.6.1
gevent==24.2.1
azure-identity==1.17.1
werkzeug=2.2.2

Is there a known workaround for this? I have tried various alternatives, to no avail.
e.g.

  • passing a loop to the filesystem init, then calling _ls
  • calling asyncio.run(fs.ls)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the minimal example in server.py, especially the listing view and its AzureBlobFileSystem.ls call, then reproduce it with the versions in requirements.txt. Read the fsspec/asyn.py sync traceback and compare the gevent and threading configurations. Done means identifying and documenting a supported workaround or a narrowly scoped fix for synchronous filesystem calls inside this setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, flask, python
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.