fsspec / fsspec/filesystem_spec
ArrowFSWrapper should not use "/" root_marker for all filesystems
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
For the arrow S3FileSystem for example, using root_market = "/" throws an error when trying to upload a local file. Love the arrow integration, please let me know if I'm misusing something here.
from pyarrow.fs import S3FileSystem
from fsspec.implementations.arrow import ArrowFSWrapper
fs = ArrowFSWrapper(S3FileSystem())
fs.upload('foo.txt', 'bucket/folder/foo.txt')
Traceback (most recent call last):
File "", line 1, in
File ".../fsspec/spec.py", line 1530, in upload
return self.put(lpath, rpath, recursive=recursive, **kwargs)
File ".../fsspec/spec.py", line 1057, in put
self.put_file(lpath, rpath, **kwargs)
File ".../fsspec/spec.py", line 984, in put_file
self.mkdirs(self._parent(os.fspath(rpath)), exist_ok=True)
File ".../fsspec/spec.py", line 1498, in mkdirs
return self.makedirs(path, exist_ok=exist_ok)
File ".../fsspec/implementations/arrow.py", line 22, in wrapper
return func(*args, **kwargs)
File ".../fsspec/implementations/arrow.py", line 193, in makedirs
self.fs.create_dir(path, recursive=True)
File "pyarrow/_fs.pyx", line 603, in pyarrow._fs.FileSystem.create_dir
File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Path cannot start with a separator ('/bucket/folder')
If I set root_market to '' manually, the upload succeeds
fs.__class__.root_marker = ''
fs.upload('foo.txt', 'bucket/folder/foo.txt')
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with fsspec/implementations/arrow.py, especially ArrowFSWrapper.root_marker and makedirs, and reproduce the upload example with pyarrow's S3FileSystem. Compare the wrapper's path handling with the filesystem's accepted paths; done means uploads no longer prepend an invalid separator while other Arrow filesystems continue to work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100