fsspec / fsspec/adlfs

Usage of `weakref.finalize` causes `RuntimeError: Loop is not running`

Open
#495 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
211
Forks
120
PR merge metrics
No merged PRs in 30d

Description

import pytest
from adlfs import AzureBlobFileSystem


@pytest.fixture()
def fs():
    return AzureBlobFileSystem(..., asynchronous=True)


@pytest.mark.asyncio
async def test_something(fs: AzureBlobFileSystem):
    pass

A simple setup like this, where the fs is created within an async context (event loop will be set up by pytest-asyncio here), will result in a RuntimeError: Loop is not running during gc.

This is caused by these lines:

https://github.com/fsspec/adlfs/blob/adb9c53b74a0d420625b86dd00fbe615b43201d2/adlfs/spec.py#L346-L349

which obtain a reference to the current event loop, however, since there's no guarantee when finalization on the refs will run, this can also happen after the referenced event loop has been closed.


Since all they're doing is closing things, I would like to propose adding a close method to the implementation, which can be manually invoked to do proper cleanup, and a conditional check on the finalizers, so they may not run if the resources have already been closed.

Then, all that's needed in the code to fix the issue is something like

@pytest.fixture()
async def fs():
    fs = AzureBlobFileSystem(..., asynchronous=True)
    yield fs
    await fs.close()

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in adlfs/spec.py around lines 346-349 and trace how weakref.finalize obtains and uses the event loop. Review the issue's proposed explicit close flow and fixture example. Done means resources can be closed explicitly and garbage-collection finalization does not raise RuntimeError after the loop has closed.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
backend, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.