fsspec / fsspec/filesystem_spec
API: unclear specification of mkdir behaviour when directory already exists
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
The documentation of the mkdir method (of the base class) is not explicit about this (https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.mkdir):
Create directory entry at path
For systems that don't have true directories, may create an for
this instance only and not touch the real filesystem
Parameters
----------
path: str
location
create_parents: bool
if True, this is equivalent to ``makedirs``
So this doesn't explicitly mention if this method is expected to work or fail it the directory already exists.
It does mention that with create_parents=True, it is equivalent to makedirs, however. And makedirs on the other hand has a clear keyword about this with a default of exist_ok=False (https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.makedirs). Given that it says it is "equivalent", I could have assumed mkdir(.., create_parents=True) to fail on existing directories.
But, when looking at some of the implementations of mkdir, this is not what happens in practice. Eg for the LocalFileSystem:
So you clearly see here that in case of create_parents=True, a exist_ok=True is passed to makedirs, overriding its default of False.
The same for the in-memory filesystem (it doesn't raise if the path was already present in the list of pseudo_dirs).
But for example the Azure Blob filesystem doest not do this: https://github.com/dask/adlfs/issues/134
So questions that I see:
- What should be the expected behaviour in the specification?
- If
mkdirshould fail on existing directories (so following the default ofmakedirs, is there a way to create a directory without failing? (except for manually checking first if it already exists or not?)
For the last one, the answer could be to use makedirs instead, but it seems mkdir is more guaranteed to exist (not all FileSystems seem to consistently implement makedirs, while mkdir always seems to be present).
For the fsspec-wrapper in pyarrow, we are using mkdir (and I did that with the assumption it to work when the directory already exists, as that is the behaviour we need, and that's what I saw when testing with the local filesystem)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the AbstractFileSystem.mkdir and makedirs specifications in the linked API documentation, then compare the LocalFileSystem and in-memory mkdir implementations with the Azure Blob behavior linked in the issue. Resolve the expected existing-directory semantics and document the supported way to request each behavior; done requires an agreed specification that implementations can follow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, documentation
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100