fsspec / fsspec/filesystem_spec

"RuntimeError: Loop is not running" when combining ZipFileSystem and DropboxDriveFileSystem

Open
#844 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

We (myself and @annieleal) are trying to read a zip file from dropbox (see https://github.com/annieleal/rces-final-project/issues/1 for more details).

We are not able to get it to work. This is what we have tried

import dropbox
import dropboxdrivefs as dbx
import fsspec
from fsspec.implementations.zip import ZipFileSystem

token = 'u19t2wI3eH4AAAAAAAAAAdplJXSWJIRd1Rp8sAl1MIcxTxz4fkJ8xP_y8dqS7sdv'
path = '/RN-20211117121141_CF2D9BA9D51D3B7CE0538486ABC0F5FD/idp2021/GEOTRACES_IDP2021_v1_seawater_netcdf.zip'

dfs = dbx.DropboxDriveFileSystem(token=token)
dbox_file = dfs.open(path)
zfs = ZipFileSystem(dbox_file)

# does the same thing - in one pass
zfs = ZipFileSystem(path, target_protocol='dropbox', target_options={'token': token})

This gives the following error

INFO:Request to files/get_temporary_link
INFO:Request to files/get_metadata
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_353/2838041811.py in <module>
     10 dfs = dbx.DropboxDriveFileSystem(token=token)
     11 dbox_file = dfs.open(path)
---> 12 zfs = ZipFileSystem(dbox_file)

~/.local/lib/python3.8/site-packages/fsspec/spec.py in __call__(cls, *args, **kwargs)
     72             return cls._cache[token]
     73         else:
---> 74             obj = super().__call__(*args, **kwargs)
     75             # Setting _fs_token here causes some static linters to complain.
     76             obj._fs_token_ = token

~/.local/lib/python3.8/site-packages/fsspec/implementations/zip.py in __init__(self, fo, mode, target_protocol, target_options, block_size, **kwargs)
     55             fo = files[0]
     56         self.fo = fo.__enter__()  # the whole instance is a context
---> 57         self.zip = zipfile.ZipFile(self.fo)
     58         self.block_size = block_size
     59         self.dir_cache = None

/srv/conda/envs/notebook/lib/python3.8/zipfile.py in __init__(self, file, mode, compression, allowZip64, compresslevel, strict_timestamps)
   1267         try:
   1268             if mode == 'r':
-> 1269                 self._RealGetContents()
   1270             elif mode in ('w', 'x'):
   1271                 # set the modified flag so central directory gets written

/srv/conda/envs/notebook/lib/python3.8/zipfile.py in _RealGetContents(self)
   1330         fp = self.fp
   1331         try:
-> 1332             endrec = _EndRecData(fp)
   1333         except OSError:
   1334             raise BadZipFile("File is not a zip file")

/srv/conda/envs/notebook/lib/python3.8/zipfile.py in _EndRecData(fpin)
    272     except OSError:
    273         return None
--> 274     data = fpin.read()
    275     if (len(data) == sizeEndCentDir and
    276         data[0:4] == stringEndArchive and

~/.local/lib/python3.8/site-packages/fsspec/implementations/http.py in read(self, length)
    496         else:
    497             length = min(self.size - self.loc, length)
--> 498         return super().read(length)
    499 
    500     async def async_fetch_all(self):

~/.local/lib/python3.8/site-packages/fsspec/spec.py in read(self, length)
   1467             # don't even bother calling fetch
   1468             return b""
-> 1469         out = self.cache._fetch(self.loc, self.loc + length)
   1470         self.loc += len(out)
   1471         return out

~/.local/lib/python3.8/site-packages/fsspec/caching.py in _fetch(self, start, end)
    374         ):
    375             # First read, or extending both before and after
--> 376             self.cache = self.fetcher(start, bend)
    377             self.start = start
    378         elif start < self.start:

~/.local/lib/python3.8/site-packages/fsspec/asyn.py in wrapper(*args, **kwargs)
     85     def wrapper(*args, **kwargs):
     86         self = obj or args[0]
---> 87         return sync(self.loop, func, *args, **kwargs)
     88 
     89     return wrapper

~/.local/lib/python3.8/site-packages/fsspec/asyn.py in sync(loop, func, timeout, *args, **kwargs)
     43     # and we will wait for it
     44     if loop is None or loop.is_closed():
---> 45         raise RuntimeError("Loop is not running")
     46     try:
     47         loop0 = grl()

RuntimeError: Loop is not running

Any ideas?

I'm not sure if this is a dropboxfs-specific problem or a broader integration issues, which is why I have posted it here first.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the example with ZipFileSystem and DropboxDriveFileSystem, then inspect the fsspec implementations/zip.py and fsspec/asyn.py paths shown in the traceback. Determine why the Dropbox-backed read reaches a non-running event loop; done means the demonstrated zip read works without RuntimeError and the integration behavior is covered by an appropriate regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.