ipfs / ipfs/kubo

Properly Handling Filesystems

Open
#1,642 6 comments 0 reactions 0 assignees View on GitHub
need/community-input
Dominant language
Go
Stars
17.1k
Forks
3.2k
Avg merge
3d 18h
Merged PRs (30d)
11

Description

So far, we've only been handling "regular files" and "directories". We've recently added symlink support (#1627, thanks @whyrusleeping). Named pipes and other special files are still not supported and continue to hang (#1072).
- ipfs shouldn't hang
- we should decide how to handle them.

i think git ignores everything else. But since ipfs is more general than git (i.e. we may store filesystems representing containers or entire VMs) we must go through all special files and decide how to handle them. (I suspect that -- to properly represent containers and VMs -- we will need to be able to ingest + reproduce them all). Worth mentioning that we should be able to handle other non-UNIX platform-specific things in an extensible way, like solaris Doors, xattrs in linux/bsd/aix/osx, and so on.

I suggest the following process:
- [ ] enumerate all filesystem files + attributes that we may want to track
- defer most things (like MIME types, xattrs, etc) to the "Metadata" object
- [ ] suggest patches to unixfs that will address the requirements
- [ ] select one + implement it

The standard file types in UNIX are (am i missing any?):
- [x] Regular files. added as unixfs `File` type.
- [x] Directories. added as unixfs `Dir` types.
- [x] Symlinks. added as unixfs `Symlink` type
- [ ] Named pipe. (unixplex pipe)
- [ ] (Unix) Sockets. (duplex pipe)
- [ ] Device files.

Non-UNIX but widely used:
- [ ] [Doors](https://en.wikipedia.org/wiki/Doors_%28computing%29) in Solaris
- [ ] [xattrs](https://en.wikipedia.org/wiki/Extended_file_attributes) in Linux/BSD/OSX/AIX/...
- [ ] whatever windows does
### Transparent resolution across object shards

This is not super relevant, but comes to bear here. We will have to teach tools to transparently resolve across shards. This is already done for "chunked files" (i.e. files that are "large" and thus sharded across multiple objects). And we will have to do it for "chunked dirs" too (not yet supported). (Other datastructures, like keychain's `Signatures` will need similar resolving as well).

FWIW, most git tools do this transparent resolution for the `commit.tree` pairing-- that is, commits have one tree, so referencing `/foo/bar` really means `/tree/foo/bar` in most git tools, including github.

cc @whyrusleeping @rht @wking @tv42 @mildred

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.