fsspec / fsspec/filesystem_spec

fsmap can't 'get' on simplecache wrapped on adlfs

Open
#794 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

fs = fsspec.filesystem('simplecache', fs=fsspec.filesystem('az', **kw),)
fsm = fs.get_mapper('/some/where')

In [130]: next(iter(fsm.values()))
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-130-88122b9ca828> in <module>
----> 1 next(iter(fsm.values()))

~\programfiles\miniconda\envs\fixtures-cache\lib\_collections_abc.py in __iter__(self)
    867     def __iter__(self):
    868         for key in self._mapping:
--> 869             yield self._mapping[key]
    870
    871

~\programfiles\miniconda\envs\fixtures-cache\lib\site-packages\fsspec\mapping.py in __getitem__(self, key, default)
    133         k = self._key_to_str(key)
    134         try:
--> 135             result = self.fs.cat(k)
    136         except self.missing_exceptions:
    137             if default is not None:

~\programfiles\miniconda\envs\fixtures-cache\lib\site-packages\fsspec\implementations\cached.py in <lambda>(*args, **kw)
    391             # all the methods defined in this class. Note `open` here, since
    392             # it calls `_open`, but is actually in superclass
--> 393             return lambda *args, **kw: getattr(type(self), item).__get__(self)(
    394                 *args, **kw
    395             )

~\programfiles\miniconda\envs\fixtures-cache\lib\site-packages\fsspec\implementations\cached.py in cat(self, path, recursive, on_error, callback, **kwargs)
    568             fns.append(fn)
    569         if getpaths:
--> 570             self.fs.get(getpaths, storepaths)
    571             self.save_cache()
    572

~\programfiles\miniconda\envs\fixtures-cache\lib\site-packages\fsspec\asyn.py in wrapper(*args, **kwargs)
     86     def wrapper(*args, **kwargs):
     87         self = obj or args[0]
---> 88         return sync(self.loop, func, *args, **kwargs)
     89
     90     return wrapper

~\programfiles\miniconda\envs\fixtures-cache\lib\site-packages\fsspec\asyn.py in sync(loop, func, timeout, *args, **kwargs)
     67         raise FSTimeoutError
     68     if isinstance(result[0], BaseException):
---> 69         raise result[0]
     70     return result[0]
     71

~\programfiles\miniconda\envs\fixtures-cache\lib\site-packages\fsspec\asyn.py in _runner(event, coro, result, timeout)
     23         coro = asyncio.wait_for(coro, timeout=timeout)
     24     try:
---> 25         result[0] = await coro
     26     except Exception as ex:
     27         result[0] = ex

~\programfiles\miniconda\envs\fixtures-cache\lib\site-packages\fsspec\asyn.py in _get(self, rpath, lpath, recursive, callback, **kwargs)
    422         from fsspec.implementations.local import make_path_posix
    423
--> 424         rpath = self._strip_protocol(rpath)
    425         lpath = make_path_posix(lpath)
    426         rpaths = await self._expand_path(rpath, recursive=recursive)

~\programfiles\miniconda\envs\fixtures-cache\lib\site-packages\adlfs\spec.py in _strip_protocol(cls, path)
    454         STORE_SUFFIX = ".dfs.core.windows.net"
    455         logger.debug(f"_strip_protocol for {path}")
--> 456         if not path.startswith(("abfs://", "az://", "abfss://")):
    457             path = path.lstrip("/")
    458             path = "abfs://" + path

AttributeError: 'list' object has no attribute 'startswith'

AzureDataLakeFS is derived from AsyncFileSystem. Somewhere WholeFileCache creates a list of paths to get by the wrapped fs self.fs.get. But AsyncFilesystem's _get is only expecting one path.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the WholeFileCache implementation in fsspec/implementations/cached.py and the AsyncFileSystem _get path in fsspec/asyn.py. Reproduce the simplecache-wrapped AzureDataLakeFS case, then inspect how the wrapped filesystem receives multiple paths. Done means the mapper value lookup completes without the list-path AttributeError; add or update a regression test if the repository identifies a suitable test location.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.