mhx / mhx/dwarfs

Add support for normalizing / hiding hardlinks in the output image

Open
#349 6 comments 0 reactions 0 assignees View on GitHub
enhancement wishlist
Dominant language
C++
Stars
2.6k
Forks
92
PR merge metrics
No merged PRs in 30d

Description

At the moment, `mkdwarfs` faithfully reflects the hardlink relation between files from the source file-system (i.e. if file `/a` and `/b` are hardlinks to one-another, so they appear in the `image.dwarfs` when mounted with FUSE). (And this is great.)

However, there are use-cases when one wants to break (or tweak) this relationship in the resulting image, namely:
* (A) either having all files appear with exactly 1 hardlink, although underneath in the resulting image they share exactly the same content;
* (B) either having all duplicate files (based on the hash) in the source file-system appear as hard-links in the resulting image;

----

Why would case (A) (i.e. hardlinks are dissolved) be useful? The hardlinks in the source file-system might be an implementation detail / optimization, and the user wouldn't want that fact shown in the resulting image.

For example there are techniques of using `rsync` to create snapshots back-ups that use hardlinks. However, the user might want to create a final image, and would like to hide this fact away.

Why would case (B) (i.e. all duplicate files to become hard-links to each-other), some other times, it's beneficial to have deduplicated files share the same hard-links, as this would allow more advanced tools to take benefit of this information in how they operate. (Again, `rsync` might be one such tool.)

----

As a side-note to point (B), how does DwarFS handle files that have more than 65536 hard-links?

(Where would one encounter such a case: MAME supporting files, namely cheats which are XML files, and for some reason, the developers of MAME decided to copy the same hash-duplicate file ~120K times in the same folder...)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the mkdwarfs path that preserves source hardlink relationships and review how the resulting image exposes them when mounted with FUSE. Compare the requested modes for dissolving hardlinks and deduplicating identical files, and investigate the stated question about files with more than 65536 hardlinks. Done means the intended behavior and limits are defined and covered by appropriate tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
operating-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.