fsspec / fsspec/filesystem_spec
fsmap can't 'get' on simplecache wrapped on adlfs
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
fs = fsspec.filesystem('simplecache', fs=fsspec.filesystem('az', **kw),)
fsm = fs.get_mapper('/some/where')
In [130]: next(iter(fsm.values()))
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-130-88122b9ca828> in <module>
----> 1 next(iter(fsm.values()))
~\programfiles\miniconda\envs\fixtures-cache\lib\_collections_abc.py in __iter__(self)
867 def __iter__(self):
868 for key in self._mapping:
--> 869 yield self._mapping[key]
870
871
~\programfiles\miniconda\envs\fixtures-cache\lib\site-packages\fsspec\mapping.py in __getitem__(self, key, default)
133 k = self._key_to_str(key)
134 try:
--> 135 result = self.fs.cat(k)
136 except self.missing_exceptions:
137 if default is not None:
~\programfiles\miniconda\envs\fixtures-cache\lib\site-packages\fsspec\implementations\cached.py in <lambda>(*args, **kw)
391 # all the methods defined in this class. Note `open` here, since
392 # it calls `_open`, but is actually in superclass
--> 393 return lambda *args, **kw: getattr(type(self), item).__get__(self)(
394 *args, **kw
395 )
~\programfiles\miniconda\envs\fixtures-cache\lib\site-packages\fsspec\implementations\cached.py in cat(self, path, recursive, on_error, callback, **kwargs)
568 fns.append(fn)
569 if getpaths:
--> 570 self.fs.get(getpaths, storepaths)
571 self.save_cache()
572
~\programfiles\miniconda\envs\fixtures-cache\lib\site-packages\fsspec\asyn.py in wrapper(*args, **kwargs)
86 def wrapper(*args, **kwargs):
87 self = obj or args[0]
---> 88 return sync(self.loop, func, *args, **kwargs)
89
90 return wrapper
~\programfiles\miniconda\envs\fixtures-cache\lib\site-packages\fsspec\asyn.py in sync(loop, func, timeout, *args, **kwargs)
67 raise FSTimeoutError
68 if isinstance(result[0], BaseException):
---> 69 raise result[0]
70 return result[0]
71
~\programfiles\miniconda\envs\fixtures-cache\lib\site-packages\fsspec\asyn.py in _runner(event, coro, result, timeout)
23 coro = asyncio.wait_for(coro, timeout=timeout)
24 try:
---> 25 result[0] = await coro
26 except Exception as ex:
27 result[0] = ex
~\programfiles\miniconda\envs\fixtures-cache\lib\site-packages\fsspec\asyn.py in _get(self, rpath, lpath, recursive, callback, **kwargs)
422 from fsspec.implementations.local import make_path_posix
423
--> 424 rpath = self._strip_protocol(rpath)
425 lpath = make_path_posix(lpath)
426 rpaths = await self._expand_path(rpath, recursive=recursive)
~\programfiles\miniconda\envs\fixtures-cache\lib\site-packages\adlfs\spec.py in _strip_protocol(cls, path)
454 STORE_SUFFIX = ".dfs.core.windows.net"
455 logger.debug(f"_strip_protocol for {path}")
--> 456 if not path.startswith(("abfs://", "az://", "abfss://")):
457 path = path.lstrip("/")
458 path = "abfs://" + path
AttributeError: 'list' object has no attribute 'startswith'
AzureDataLakeFS is derived from AsyncFileSystem. Somewhere WholeFileCache creates a list of paths to get by the wrapped fs self.fs.get. But AsyncFilesystem's _get is only expecting one path.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the WholeFileCache implementation in fsspec/implementations/cached.py and the AsyncFileSystem _get path in fsspec/asyn.py. Reproduce the simplecache-wrapped AzureDataLakeFS case, then inspect how the wrapped filesystem receives multiple paths. Done means the mapper value lookup completes without the list-path AttributeError; add or update a regression test if the repository identifies a suitable test location.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100