fsspec / fsspec/filesystem_spec

API: unclear specification of mkdir behaviour when directory already exists

Open
#479 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

The documentation of the mkdir method (of the base class) is not explicit about this (https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.mkdir):

        Create directory entry at path

        For systems that don't have true directories, may create an for
        this instance only and not touch the real filesystem

        Parameters
        ----------
        path: str
            location
        create_parents: bool
            if True, this is equivalent to ``makedirs``

So this doesn't explicitly mention if this method is expected to work or fail it the directory already exists.

It does mention that with create_parents=True, it is equivalent to makedirs, however. And makedirs on the other hand has a clear keyword about this with a default of exist_ok=False (https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.makedirs). Given that it says it is "equivalent", I could have assumed mkdir(.., create_parents=True) to fail on existing directories.

But, when looking at some of the implementations of mkdir, this is not what happens in practice. Eg for the LocalFileSystem:

https://github.com/intake/filesystem_spec/blob/e6596825b5df0d43a7590ff82f50f2b1fead3ab3/fsspec/implementations/local.py#L31-L36

So you clearly see here that in case of create_parents=True, a exist_ok=True is passed to makedirs, overriding its default of False.
The same for the in-memory filesystem (it doesn't raise if the path was already present in the list of pseudo_dirs).

But for example the Azure Blob filesystem doest not do this: https://github.com/dask/adlfs/issues/134

So questions that I see:

  • What should be the expected behaviour in the specification?
  • If mkdir should fail on existing directories (so following the default of makedirs, is there a way to create a directory without failing? (except for manually checking first if it already exists or not?)

For the last one, the answer could be to use makedirs instead, but it seems mkdir is more guaranteed to exist (not all FileSystems seem to consistently implement makedirs, while mkdir always seems to be present).
For the fsspec-wrapper in pyarrow, we are using mkdir (and I did that with the assumption it to work when the directory already exists, as that is the behaviour we need, and that's what I saw when testing with the local filesystem)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the AbstractFileSystem.mkdir and makedirs specifications in the linked API documentation, then compare the LocalFileSystem and in-memory mkdir implementations with the Azure Blob behavior linked in the issue. Resolve the expected existing-directory semantics and document the supported way to request each behavior; done requires an agreed specification that implementations can follow.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, documentation
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.