fsspec / fsspec/filesystem_spec
Unsupported 'cache_type' argument passed on to PyArrow's HDFS classes from WholeFileCacheFileSystem
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
Hi,
In fsspec 0.5.2 via Conda, it appears the fsspec.implementations.cached.WholeFileCacheFileSystem might be feeding unwanted arguments to the PyArrow lib methods, such as cache_type.
If I setup my cached Filesystem like so:
import fsspec
fs = fsspec.filesystem(
protocol="filecache",
target_protocol='hdfs',
cache_storage='/mylocalfolder',
cache_check=3600, #one hour, Number of seconds between reload of cache metadata
check_files=True, #doesn't work with HTTP files
expiry_time=False, #The time in seconds after which a local copy is considered useless. Set to falsy to prevent expiry. 604800 => one week
target_options={'path': '/myhdfspath/', 'host': 'myhdfshost'}
)
This works fine:
fs.walk('/adir')
But this does not:
of = fs.open('/adir/afile.txt')
with of as f:
f
I get this error
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-9-f4b5281b24bb> in <module>
----> 3 of = fs.open('/adir/afile.txt')
4 with of as f:
5 f
~/conda/envs/myenv/lib/python3.7/site-packages/fsspec/implementations/cached.py in <lambda>(*args, **kw)
259 # all the methods defined in this class. Note `open` here, since
260 # it calls `_open`, but is actually in superclass
--> 261 return lambda *args, **kw: getattr(type(self), item)(self, *args, **kw)
262 if item == "__class__":
263 return type(self)
~/conda/envs/myenv/lib/python3.7/site-packages/fsspec/spec.py in open(self, path, mode, block_size, **kwargs)
706 ac = kwargs.pop("autocommit", not self._intrans)
707 f = self._open(
--> 708 path, mode=mode, block_size=block_size, autocommit=ac, **kwargs
709 )
710 if not ac:
~/conda/envs/myenv/lib/python3.7/site-packages/fsspec/implementations/cached.py in <lambda>(*args, **kw)
259 # all the methods defined in this class. Note `open` here, since
260 # it calls `_open`, but is actually in superclass
--> 261 return lambda *args, **kw: getattr(type(self), item)(self, *args, **kw)
262 if item == "__class__":
263 return type(self)
~/conda/envs/myenv/lib/python3.7/site-packages/fsspec/implementations/cached.py in _open(self, path, mode, **kwargs)
333
334 # call target filesystems open
--> 335 f = self.fs._open(path, **kwargs)
336 with open(fn, "wb") as f2:
337 if f.blocksize and f.size:
~/conda/envs/myenv/lib/python3.7/site-packages/fsspec/implementations/hdfs.py in <lambda>(*args, **kw)
114 # all the methods defined in this class. Note `open` here, since
115 # it calls `_open`, but is actually in superclass
--> 116 return lambda *args, **kw: getattr(PyArrowHDFS, item)(self, *args, **kw)
117 if item == "__class__":
118 return PyArrowHDFS
~/conda/envs/myenv/lib/python3.7/site-packages/fsspec/implementations/hdfs.py in _open(self, path, mode, block_size, autocommit, **kwargs)
70 if not autocommit:
71 raise NotImplementedError
---> 72 return HDFSFile(self, path, mode, block_size, **kwargs)
73
74 def __reduce_ex__(self, protocol):
~/conda/envs/myenv/lib/python3.7/site-packages/fsspec/implementations/hdfs.py in __init__(self, fs, path, mode, block_size, **kwargs)
169 self.mode = mode
170 self.block_size = block_size
--> 171 self.fh = fs.pahdfs.open(path, mode, block_size, **kwargs)
172 if self.fh.readable():
173 self.seek_size = self.size()
~/conda/envs/myenv/lib/python3.7/site-packages/pyarrow/io-hdfs.pxi in pyarrow.lib.HadoopFileSystem.open()
TypeError: open() got an unexpected keyword argument 'cache_type'
I get a similar error for get().
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with implementations/cached.py and implementations/hdfs.py, following the _open call shown in the traceback. Reproduce the filecache setup against HDFS and compare open() and get() argument forwarding. Done means WholeFileCacheFileSystem no longer sends unsupported cache_type arguments to PyArrow HDFS methods and both operations work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100