fsspec / fsspec/gcsfs

isdir/info method works incorrectly

Open
#574 20 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
398
Forks
181
Avg merge
2d 10h
Merged PRs (30d)
36

Description

Hello,
I've found a strange behavior of the isdir method (digging deeper also with info method). It returns incorrect values. These values seem to be returned randomly.

I use Python 3.10.12 and I've tested this behavior on gscfs=2022.3.0, and the latest version gscfs=2023.6.0

I've prepared a helper function to show what is happening here:

from gcsfs import GCSFileSystem

fs = GCSFileSystem()


def check_is_dir(path):
    is_dir = fs.isdir(path)
    info_type = fs.info(path)["type"]

    print(path, is_dir, info_type)

Problem example

An exemplary run:

check_is_dir('gs://my/super')
check_is_dir('gs://my/super/secret')
check_is_dir('gs://my/super/secret/gcs')
check_is_dir('gs://my/super/secret/gcs/directory')
check_is_dir('gs://my/super/secret/gcs/directory/file.json')

Results:

gs://my/super False director  # the first string is a path, the first boolean is a value returned by isdir method, and the second string is 'type' value in the dictionary returned by fs.info(path)
gs://my/super/secret True directory
gs://my/super/secret/gcs False directory
gs://my/super/secret/gcs/directory True directory
gs://my/super/secret/gcs/directory/file.json False file

As you can see, some directories are incorrectly treated as files. So more, values returned by the info and isdir methods are inconsistent.

Another insight

Changing the order of calling these methods, like in the snippet below:

def check_is_dir(path):
    info_type = fs.info(path)["type"]
    is_dir = fs.isdir(path)

    print(path, is_dir, info_type)

makes is_dir contains a correct value, but info_type incorrect one. Like here:

gs://my/super True file  # the first string is a path, the first boolean is a value returned by isdir method, and the second string is 'type' value in the dictionary returned by fs.info(path)
gs://my/super/secret True directory
gs://my/super/secret/gcs True file
gs://my/super/secret/gcs/directory True directory
gs://my/super/secret/gcs/directory/file.json False file

Workaround

For now, my workaround is to run isdir method two times:

def check_is_dir(path):
    is_dir = fs.isdir(path)
    is_dir = fs.isdir(path)
    info_type = fs.info(path)["type"]

    print(path, is_dir, info_type)

It works:

gs://my/super True directory  # the first string is a path, the first boolean is a value returned by isdir method, and the second string is 'type' value in the dictionary returned by fs.info(path)
gs://my/super/secret True directory
gs://my/super/secret/gcs True directory
gs://my/super/secret/gcs/directory True directory
gs://my/super/secret/gcs/directory/file.json False file

But I want to work with this library without such workaround ;)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the GCSFileSystem.isdir and info methods described in the issue, then run the provided check_is_dir helper against the listed paths using the reported Python and gcsfs versions. Compare the results when the methods are called in each order; done means both methods consistently identify directories and files without requiring repeated calls.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.