fsspec / fsspec/filesystem_spec

ArrowFSWrapper should not use "/" root_marker for all filesystems

Open
#1,464 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

For the arrow S3FileSystem for example, using root_market = "/" throws an error when trying to upload a local file. Love the arrow integration, please let me know if I'm misusing something here.

from pyarrow.fs import S3FileSystem
from fsspec.implementations.arrow import ArrowFSWrapper

fs = ArrowFSWrapper(S3FileSystem())
fs.upload('foo.txt', 'bucket/folder/foo.txt')

Traceback (most recent call last):
File "", line 1, in
File ".../fsspec/spec.py", line 1530, in upload
return self.put(lpath, rpath, recursive=recursive, **kwargs)
File ".../fsspec/spec.py", line 1057, in put
self.put_file(lpath, rpath, **kwargs)
File ".../fsspec/spec.py", line 984, in put_file
self.mkdirs(self._parent(os.fspath(rpath)), exist_ok=True)
File ".../fsspec/spec.py", line 1498, in mkdirs
return self.makedirs(path, exist_ok=exist_ok)
File ".../fsspec/implementations/arrow.py", line 22, in wrapper
return func(*args, **kwargs)
File ".../fsspec/implementations/arrow.py", line 193, in makedirs
self.fs.create_dir(path, recursive=True)
File "pyarrow/_fs.pyx", line 603, in pyarrow._fs.FileSystem.create_dir
File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Path cannot start with a separator ('/bucket/folder')

If I set root_market to '' manually, the upload succeeds

fs.__class__.root_marker = ''
fs.upload('foo.txt', 'bucket/folder/foo.txt')

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with fsspec/implementations/arrow.py, especially ArrowFSWrapper.root_marker and makedirs, and reproduce the upload example with pyarrow's S3FileSystem. Compare the wrapper's path handling with the filesystem's accepted paths; done means uploads no longer prepend an invalid separator while other Arrow filesystems continue to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.