fsspec / fsspec/filesystem_spec

HTTPFileSystem prints tracebacks if skip_instance_cache=True if called more than once

Open
#1,506 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

I have been debugging an issue for a while and finally have a small reproducible snip. Using fsspec==2023.12.2

import fsspec

some_public_url = "http://replace.me.with.a.public.url"

def read_my_data(url: str) -> None:
    for i in range(10):
        fs = fsspec.filesystem("http", skip_instance_cache=True)
        read_data = fs.read_text(url)        
        print(i, read_data)

read_my_data(some_public_url)

The output looks something like

0
1
2
Exception ignored in: <finalize object at 0x7f644203f9e0; dead>
Traceback (most recent call last):
  File "/lib/python3.10/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/lib/python3.10/site-packages/fsspec/implementations/http.py", line 125, in close_session
    sync(loop, session.close, timeout=0.1)
  File "/lib/python3.10/site-packages/fsspec/asyn.py", line 80, in sync
    raise NotImplementedError("Calling sync() from within a running loop")
NotImplementedError: Calling sync() from within a running loop
Exception ignored in: <finalize object at 0x7f644203fae0; dead>
Traceback (most recent call last):
  File "/python3.10/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/lib/python3.10/site-packages/fsspec/implementations/http.py", line 125, in close_session
    sync(loop, session.close, timeout=0.1)
  File "/python3.10/site-packages/fsspec/asyn.py", line 80, in sync
    raise NotImplementedError("Calling sync() from within a running loop")
NotImplementedError: Calling sync() from within a running loop
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f644207e2f0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f644207ee90>
3
4
5
done

The number of successes before this is error is printed is not consistent but usually after 2 or 3. This traceback is not fatal and the correct data is read but this is an ugly traceback to be printed which I do not know how to suppress.

If instead skip_instance_cache=False I do not see this issue. My guess is something wrong is cached internally between runs?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the repeated skip_instance_cache=True reproducer and inspect fsspec/implementations/http.py around close_session and fsspec/asyn.py around sync, as identified in the traceback. Check the related finalization and session lifecycle behavior; done means repeated HTTPFileSystem use no longer prints tracebacks or unclosed-session warnings while still reading the data.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.