fsspec / fsspec/filesystem_spec

hdfs `delete` fails where `rm` succeeds. Also `maxdepth` is unspported.

Open
#528 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

good first issue
Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

In the spec, delete is supposed to be an alias for rm. However this fails for hdfs.

fs = fsspec.filesystem("hdfs")
fs.delete("hdfs:///user/f4hy/test/testfile.txt")
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-16-9390bf0d502b> in <module>
----> 1 fs.delete("hdfs:///user/bfahy2/bin/test2.py")

~/.local/lib/python3.7/site-packages/fsspec/spec.py in delete(self, path, recursive, maxdepth)
   1136     def delete(self, path, recursive=False, maxdepth=None):
   1137         """Alias of :ref:`FilesystemSpec.rm`."""
-> 1138         return self.rm(path, recursive=recursive, maxdepth=maxdepth)
   1139 
   1140     def upload(self, lpath, rpath, recursive=False, **kwargs):

TypeError: rm() got an unexpected keyword argument 'maxdepth'

while rm works fine, but only without maxdepth

fs.rm("hdfs:///user/f4hy/test/testfile.txt") # ok
fs.rm("hdfs:///user/f4hy/test/testfile.txt", maxdepth=1) # errors

The pyarrow implementation is just passing rm to the pyarrow filespec, but it doesnt exactly match up with fsspec.

We should make sure all options in the fsspec are supported, or if not at at least spew a better error.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the FilesystemSpec.delete method shown in spec.py and trace the HDFS and PyArrow rm implementations. Check how recursive and maxdepth are handled for both rm and delete, then verify that supported options behave consistently or produce a clear error when unsupported.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.