fsspec / fsspec/gcsfs

sys:1: RuntimeWarning: coroutine 'GCSFileSystem._info' was never awaited

Open
#285 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
398
Forks
181
Avg merge
2d 10h
Merged PRs (30d)
36

Description

What happened:
Exception:

Traceback (most recent call last):
  File "googleload.py", line 12, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File "/path/to/python/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "googleload.py", line 9, in main
    async with await fs.open("/path/to/file.txt") as fp:
  File "/path/to/python/lib/python3.8/site-packages/fsspec/spec.py", line 838, in open
    f = self._open(
  File "/path/to/python/lib/python3.8/site-packages/gcsfs/core.py", line 1141, in _open
    return GCSFile(
  File "/path/to/python/lib/python3.8/site-packages/gcsfs/core.py", line 1265, in __init__
    super().__init__(
  File "/path/to/python/lib/python3.8/site-packages/fsspec/spec.py", line 1135, in __init__
    self.size = self.details["size"]
TypeError: 'coroutine' object is not subscriptable
ERROR:asyncio:Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x10a6ad6a0>
sys:1: RuntimeWarning: coroutine 'GCSFileSystem._info' was never awaited

What you expected to happen:
I'm not 100% sure how to use the new async part, as it is quite poorly documented so far.
I don't even know which part of open/read/with needs to be awaited/asynced.
But no matter what I choose there, it fails because the exception already fails in fs.open.

I suspect that self.details is set to the return value of some function, and in the original sync code the return value was just the details.
As you can see in line 1134 of fsspec's specy.py:

self.details = fs.info(path)

I'm guessing that fs.info is a coroutine in the async implementation, therefore assigns an Awaitable to self.details.

Minimal Complete Verifiable Example:

import asyncio
import gcsfs

async def main():
    loop = asyncio.get_event_loop()
    fs = gcsfs.GCSFileSystem(project="my-project", asynchronous=True, loop=loop)
    await fs.set_session()

    async with await fs.open("/path/to/file.txt") as fp:
        print(await fp.read())

asyncio.get_event_loop().run_until_complete(main())

What I would like to know:
Could you also provide me with the way how the opened file-pointer is supposed to be used afterwards?
Unfortunately, you don't have any documentation beyond that I need to await the session in the beginning.

Do I await on fs.open, or is fs.open itself sync? Do I await fp.read or is everything buffered on open already anyway and the actual read can be sync because it just reads from memory?

Environment:

  • gcsfs version: 0.7.0
  • Python version: 3.8.5
  • Operating System: macOS 10.15.4
  • Install method (conda, pip, source): miniconda env, installed via pip

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with gcsfs/core.py at _open and GCSFile, then compare fsspec/spec.py where self.details is populated. Reproduce the supplied asynchronous example and determine the expected await behavior for fs.open and fp.read; done means the example no longer raises the coroutine or unclosed-session errors and the usage is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.