fsspec / fsspec/filesystem_spec

Unsupported 'cache_type' argument passed on to PyArrow's HDFS classes from WholeFileCacheFileSystem

Open
#182 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

Hi,

In fsspec 0.5.2 via Conda, it appears the fsspec.implementations.cached.WholeFileCacheFileSystem might be feeding unwanted arguments to the PyArrow lib methods, such as cache_type.
If I setup my cached Filesystem like so:

import fsspec

fs = fsspec.filesystem(
            protocol="filecache",
            target_protocol='hdfs',
            cache_storage='/mylocalfolder',
            cache_check=3600, #one hour, Number of seconds between reload of cache metadata
            check_files=True, #doesn't work with HTTP files
            expiry_time=False, #The time in seconds after which a local copy is considered useless. Set to falsy to prevent expiry. 604800 => one week
            target_options={'path': '/myhdfspath/', 'host': 'myhdfshost'}
        )

This works fine:

fs.walk('/adir')

But this does not:

of = fs.open('/adir/afile.txt')
with of as f:
    f

I get this error


---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-9-f4b5281b24bb> in <module>
----> 3 of = fs.open('/adir/afile.txt')
      4 with of as f:
      5     f
      
~/conda/envs/myenv/lib/python3.7/site-packages/fsspec/implementations/cached.py in <lambda>(*args, **kw)
    259             # all the methods defined in this class. Note `open` here, since
    260             # it calls `_open`, but is actually in superclass
--> 261             return lambda *args, **kw: getattr(type(self), item)(self, *args, **kw)
    262         if item == "__class__":
    263             return type(self)

~/conda/envs/myenv/lib/python3.7/site-packages/fsspec/spec.py in open(self, path, mode, block_size, **kwargs)
    706             ac = kwargs.pop("autocommit", not self._intrans)
    707             f = self._open(
--> 708                 path, mode=mode, block_size=block_size, autocommit=ac, **kwargs
    709             )
    710             if not ac:

~/conda/envs/myenv/lib/python3.7/site-packages/fsspec/implementations/cached.py in <lambda>(*args, **kw)
    259             # all the methods defined in this class. Note `open` here, since
    260             # it calls `_open`, but is actually in superclass
--> 261             return lambda *args, **kw: getattr(type(self), item)(self, *args, **kw)
    262         if item == "__class__":
    263             return type(self)

~/conda/envs/myenv/lib/python3.7/site-packages/fsspec/implementations/cached.py in _open(self, path, mode, **kwargs)
    333 
    334         # call target filesystems open
--> 335         f = self.fs._open(path, **kwargs)
    336         with open(fn, "wb") as f2:
    337             if f.blocksize and f.size:

~/conda/envs/myenv/lib/python3.7/site-packages/fsspec/implementations/hdfs.py in <lambda>(*args, **kw)
    114             # all the methods defined in this class. Note `open` here, since
    115             # it calls `_open`, but is actually in superclass
--> 116             return lambda *args, **kw: getattr(PyArrowHDFS, item)(self, *args, **kw)
    117         if item == "__class__":
    118             return PyArrowHDFS

~/conda/envs/myenv/lib/python3.7/site-packages/fsspec/implementations/hdfs.py in _open(self, path, mode, block_size, autocommit, **kwargs)
     70         if not autocommit:
     71             raise NotImplementedError
---> 72         return HDFSFile(self, path, mode, block_size, **kwargs)
     73 
     74     def __reduce_ex__(self, protocol):

~/conda/envs/myenv/lib/python3.7/site-packages/fsspec/implementations/hdfs.py in __init__(self, fs, path, mode, block_size, **kwargs)
    169         self.mode = mode
    170         self.block_size = block_size
--> 171         self.fh = fs.pahdfs.open(path, mode, block_size, **kwargs)
    172         if self.fh.readable():
    173             self.seek_size = self.size()

~/conda/envs/myenv/lib/python3.7/site-packages/pyarrow/io-hdfs.pxi in pyarrow.lib.HadoopFileSystem.open()

TypeError: open() got an unexpected keyword argument 'cache_type'

I get a similar error for get().

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with implementations/cached.py and implementations/hdfs.py, following the _open call shown in the traceback. Reproduce the filecache setup against HDFS and compare open() and get() argument forwarding. Done means WholeFileCacheFileSystem no longer sends unsupported cache_type arguments to PyArrow HDFS methods and both operations work.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.