isdir/info method works incorrectly
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 398
- Forks
- 181
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 36
Description
Hello,
I've found a strange behavior of the isdir method (digging deeper also with info method). It returns incorrect values. These values seem to be returned randomly.
I use Python 3.10.12 and I've tested this behavior on gscfs=2022.3.0, and the latest version gscfs=2023.6.0
I've prepared a helper function to show what is happening here:
from gcsfs import GCSFileSystem
fs = GCSFileSystem()
def check_is_dir(path):
is_dir = fs.isdir(path)
info_type = fs.info(path)["type"]
print(path, is_dir, info_type)
Problem example
An exemplary run:
check_is_dir('gs://my/super')
check_is_dir('gs://my/super/secret')
check_is_dir('gs://my/super/secret/gcs')
check_is_dir('gs://my/super/secret/gcs/directory')
check_is_dir('gs://my/super/secret/gcs/directory/file.json')
Results:
gs://my/super False director # the first string is a path, the first boolean is a value returned by isdir method, and the second string is 'type' value in the dictionary returned by fs.info(path)
gs://my/super/secret True directory
gs://my/super/secret/gcs False directory
gs://my/super/secret/gcs/directory True directory
gs://my/super/secret/gcs/directory/file.json False file
As you can see, some directories are incorrectly treated as files. So more, values returned by the info and isdir methods are inconsistent.
Another insight
Changing the order of calling these methods, like in the snippet below:
def check_is_dir(path):
info_type = fs.info(path)["type"]
is_dir = fs.isdir(path)
print(path, is_dir, info_type)
makes is_dir contains a correct value, but info_type incorrect one. Like here:
gs://my/super True file # the first string is a path, the first boolean is a value returned by isdir method, and the second string is 'type' value in the dictionary returned by fs.info(path)
gs://my/super/secret True directory
gs://my/super/secret/gcs True file
gs://my/super/secret/gcs/directory True directory
gs://my/super/secret/gcs/directory/file.json False file
Workaround
For now, my workaround is to run isdir method two times:
def check_is_dir(path):
is_dir = fs.isdir(path)
is_dir = fs.isdir(path)
info_type = fs.info(path)["type"]
print(path, is_dir, info_type)
It works:
gs://my/super True directory # the first string is a path, the first boolean is a value returned by isdir method, and the second string is 'type' value in the dictionary returned by fs.info(path)
gs://my/super/secret True directory
gs://my/super/secret/gcs True directory
gs://my/super/secret/gcs/directory True directory
gs://my/super/secret/gcs/directory/file.json False file
But I want to work with this library without such workaround ;)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the GCSFileSystem.isdir and info methods described in the issue, then run the provided check_is_dir helper against the listed paths using the reported Python and gcsfs versions. Compare the results when the methods are called in each order; done means both methods consistently identify directories and files without requiring repeated calls.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100