fsspec / fsspec/filesystem_spec
Return of info (FileInfo) is unspecified, need consistent way to detect get link information
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
The API specification for listdir and by inference also info reads:
The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:
I don't really understand the comment. "is TBD, but must be consistent across implementations" is an oxymoron. How can it be consistent between implementations when it is not specified yet? In order to increase the usefulness of the filesystem_spec API, this should be specified, imo.
Consider for example this:
import pprint
o = fsspec.open("ssh://127.0.0.1")
ol = fsspec.open("/")
print("Info of ssh:///bin")
pprint.pprint([x for x in o.fs.listdir('/', detail=True) if x['name'] == '/bin'][0])
print("Info of /bin")
pprint.pprint([x for x in ol.fs.listdir('/', detail=True) if x['name'] == '/bin'][0])
Output:
Info of ssh:///bin
{'gid': 0,
'mtime': datetime.datetime(2021, 6, 21, 9, 52, 39, tzinfo=datetime.timezone.utc),
'name': '/bin',
'size': 7,
'time': datetime.datetime(2024, 9, 19, 7, 14, 44, tzinfo=datetime.timezone.utc),
'type': 'link',
'uid': 0}
Info of /bin
{'created': 1601554848.8334851,
'destination': 'usr/bin',
'gid': 0,
'ino': 13,
'islink': True,
'mode': 41471,
'mtime': 1601554848.8334851,
'name': '/bin',
'nlink': 1,
'size': 147456,
'type': 'other',
'uid': 0}
This is not consistent between these only two tested implementations:
mtimeis datetime in the SSH backend, but a float for the local interface. This is already mentioned in #526, and probably should be a nanoseconds integer.- The SSH implementation returns
type: 'link'while the local one returnstype: 'other'and anotherislinkkey. - I need some reliable way to detect file/folder/link and also get the link target, which does not seem to be returned by either of these two implementations. I guess, this could be a separate issue?
The other implementations should also be tested. There is a nice comprehensive overview here. Code for all, but concrete examples are missing for some.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the listdir and info API specification in fsspec/archive.py, then review issue #526 and its comprehensive implementation overview. Compare the SSH and local filesystem examples for metadata types, entry types, and link targets. Done means a documented, consistent FileInfo contract that implementations can follow and callers can use reliably.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100