fsspec / fsspec/filesystem_spec
"RuntimeError: Loop is not running" when combining ZipFileSystem and DropboxDriveFileSystem
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
We (myself and @annieleal) are trying to read a zip file from dropbox (see https://github.com/annieleal/rces-final-project/issues/1 for more details).
We are not able to get it to work. This is what we have tried
import dropbox
import dropboxdrivefs as dbx
import fsspec
from fsspec.implementations.zip import ZipFileSystem
token = 'u19t2wI3eH4AAAAAAAAAAdplJXSWJIRd1Rp8sAl1MIcxTxz4fkJ8xP_y8dqS7sdv'
path = '/RN-20211117121141_CF2D9BA9D51D3B7CE0538486ABC0F5FD/idp2021/GEOTRACES_IDP2021_v1_seawater_netcdf.zip'
dfs = dbx.DropboxDriveFileSystem(token=token)
dbox_file = dfs.open(path)
zfs = ZipFileSystem(dbox_file)
# does the same thing - in one pass
zfs = ZipFileSystem(path, target_protocol='dropbox', target_options={'token': token})
This gives the following error
INFO:Request to files/get_temporary_link
INFO:Request to files/get_metadata
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
/tmp/ipykernel_353/2838041811.py in <module>
10 dfs = dbx.DropboxDriveFileSystem(token=token)
11 dbox_file = dfs.open(path)
---> 12 zfs = ZipFileSystem(dbox_file)
~/.local/lib/python3.8/site-packages/fsspec/spec.py in __call__(cls, *args, **kwargs)
72 return cls._cache[token]
73 else:
---> 74 obj = super().__call__(*args, **kwargs)
75 # Setting _fs_token here causes some static linters to complain.
76 obj._fs_token_ = token
~/.local/lib/python3.8/site-packages/fsspec/implementations/zip.py in __init__(self, fo, mode, target_protocol, target_options, block_size, **kwargs)
55 fo = files[0]
56 self.fo = fo.__enter__() # the whole instance is a context
---> 57 self.zip = zipfile.ZipFile(self.fo)
58 self.block_size = block_size
59 self.dir_cache = None
/srv/conda/envs/notebook/lib/python3.8/zipfile.py in __init__(self, file, mode, compression, allowZip64, compresslevel, strict_timestamps)
1267 try:
1268 if mode == 'r':
-> 1269 self._RealGetContents()
1270 elif mode in ('w', 'x'):
1271 # set the modified flag so central directory gets written
/srv/conda/envs/notebook/lib/python3.8/zipfile.py in _RealGetContents(self)
1330 fp = self.fp
1331 try:
-> 1332 endrec = _EndRecData(fp)
1333 except OSError:
1334 raise BadZipFile("File is not a zip file")
/srv/conda/envs/notebook/lib/python3.8/zipfile.py in _EndRecData(fpin)
272 except OSError:
273 return None
--> 274 data = fpin.read()
275 if (len(data) == sizeEndCentDir and
276 data[0:4] == stringEndArchive and
~/.local/lib/python3.8/site-packages/fsspec/implementations/http.py in read(self, length)
496 else:
497 length = min(self.size - self.loc, length)
--> 498 return super().read(length)
499
500 async def async_fetch_all(self):
~/.local/lib/python3.8/site-packages/fsspec/spec.py in read(self, length)
1467 # don't even bother calling fetch
1468 return b""
-> 1469 out = self.cache._fetch(self.loc, self.loc + length)
1470 self.loc += len(out)
1471 return out
~/.local/lib/python3.8/site-packages/fsspec/caching.py in _fetch(self, start, end)
374 ):
375 # First read, or extending both before and after
--> 376 self.cache = self.fetcher(start, bend)
377 self.start = start
378 elif start < self.start:
~/.local/lib/python3.8/site-packages/fsspec/asyn.py in wrapper(*args, **kwargs)
85 def wrapper(*args, **kwargs):
86 self = obj or args[0]
---> 87 return sync(self.loop, func, *args, **kwargs)
88
89 return wrapper
~/.local/lib/python3.8/site-packages/fsspec/asyn.py in sync(loop, func, timeout, *args, **kwargs)
43 # and we will wait for it
44 if loop is None or loop.is_closed():
---> 45 raise RuntimeError("Loop is not running")
46 try:
47 loop0 = grl()
RuntimeError: Loop is not running
Any ideas?
I'm not sure if this is a dropboxfs-specific problem or a broader integration issues, which is why I have posted it here first.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the example with ZipFileSystem and DropboxDriveFileSystem, then inspect the fsspec implementations/zip.py and fsspec/asyn.py paths shown in the traceback. Determine why the Dropbox-backed read reaches a non-running event loop; done means the demonstrated zip read works without RuntimeError and the integration behavior is covered by an appropriate regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100