Add support for following symlinks to files (and perhaps folders) and dereference them before adding to the image
- Dominant language
- C++
- Stars
- 2.6k
- Forks
- 92
- PR merge metrics
- No merged PRs in 30d
Description
It would be nice to have a flag (similar to `find -L` or `rsync --copy-links`) that makes `mkdwarfs` follow the symbolic link and resolve into the referent file (as opposed to just treating it as a symlink).
(As a minor tweak, perhaps having a separate flag that behaves like `rsync --copy-unsafe-links`, that keeps symlinks as symlinks if they point somewhere "inside" the source / image, or deference them if they lie outside the source / image.)
----
This feature might seem related to #21, but that issue is only concerned with the root source folder being a symlink. (This feature request here is referring to symlinks found inside the source folder during walking.)
Also, this feature is related, but orthogonal to, feature #349, which if combined with the "follow symlink feature", would create in the resulting image hardlinks to all files that in the source either are hard-links, perfect duplicates, or symlinks to files with exactly the same data.
----
Why would such a feature be useful?
A common pattern for archival systems is to have a fan-out content-addressable folder like `blobs`, where each file is named after its hash, like `blobs/aa/aa00123...`.
Then, to reconstruct the file-hierarchy, one could use either hardlinks, or symlinks to point into this folder.
However, hardlinks have some disadvantages that symlinks don't have, like for example:
* if the `blobs` folder is on a different mount-point, one can't use hardlinks across mount points, only symlinks; (for example the `blobs` folder might be an S3-like backed filesystem via FUSE, or perhaps a `mergerfs` backed pool of disks, or perhaps even something mounted over the network like NFS / Samba;)
* if, for extra safety, the individual blob files are `chattr +i` (i.e. immutable), then one can't hard-link to them, even the owner or root, unless they are `chattr -i` first;
* most filesystems impose a maximum limit on the number of hardlinks (Ext4 and BTRFS have it at 64K), and for some archives this limit is easily reached; (as hinted in #349, while putting together the MAME support files image, I got one file that appears ~120K times, thus breaking the hardlinks maximum number);
* tracking the symlinks in a `git` repository is far more easier than adding the blobs themselves; (the same for snapshoting only the references as a `tar` / `cpio`;)
----
Are there any workarounds that don't require changing `mkdwarfs`?
I don't see how, unless one:
* either prepares a clone of the source folder, translating the symlinks into hardlinks; (but hitting all of the above issues;)
* either one writes / uses another FUSE-based filesystem that masks the symlinks and translates the operations to the referred-to file; (but this might have a serious performance impact;)
* if `mkdwarfs` would support something like described in feature #6 (i.e. instead of walking the file-system, using a `tar` / `cpio` as source), then one could perhaps write a program to stream such an archive by translating the symlinks to proper files (perhaps with hard-links) and feed that to `mkdwarfs`;
Outside of these three options, I don't see any other alternative.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing how mkdwarfs walks source directories and currently handles symbolic links. Review the requested behavior for file links, folder links, and links outside the source, then determine the supported flag semantics and verify that the resulting image contains dereferenced files rather than symlinks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- cli, operating-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100