ipfs / ipfs/kubo

Replace original link when re-adding via --nocopy

Open
#7,465 1 comment 1 reaction 0 assignees View on GitHub
kind/enhancement topic/files
Dominant language
Go
Stars
17.1k
Forks
3.2k
Avg merge
3d 18h
Merged PRs (30d)
11

Description

Context : https://github.com/ipfs/go-ipfs/issues/7373#issuecomment-639972271

# Problem

If a file previously added with `--nocopy` is modified then re-added, there's an error.

> Error: data in file did not match. mirrors/ubuntu/dists/focal-updates/InRelease offset 0

This can be especially annoying when trying to mirror a package manager to IPFS, because often they'll have an index file (named `InRelease` for Ubuntu) that will change over time.

# Workarounds

To get around this problem, filesystem snapshots can be used. This way, one can implement the following workflow :

1. Host mirror files at `/mnt/storage/mirrors`
2. Create snapshots after downloading updates at `/mnt/storage/snapshots`
3. Add the snapshot to IPFS

Rinse and repeat. The problem with that is that the `/mnt/storage/snapshots` folder will get an ever-growing amount of snapshots to keep without any possibility to cleanup without breaking the whole mirror. That can be a bit annoying, so we need a mechanism for IPFS to be a bit more machine-friendly when using `--nocopy`.

# Solutions

* If a file added using `--nocopy` matches another one using `--nocopy`, the old link should be replaced with the new link.

This will allow the removal of obsolete filesystem snapshots without breaking the mirror.

This *could* be implemented with a `--replace` flag, but ideally, it should be the default behaviour.

* If a file added using `--nocopy` has the same path as another one but a different hash, attempt to recover the broken file via IPFS and store it *without* using `--nocopy`.

It might occur that the modification to the original file was accidental or that we want to preserve the old file to keep the old mirror going. The file could even be recoverable from the IPFS network if someone else has it. In such case, it would make sense to attempt to promote broken `--nocopy` files. The recover operation should be launched asynchronously.

This solution would allow someone to create a mirror without relying on filesystem snapshots. The problem with that is they'll periodically break/repair their mirror, which isn't ideal. It *does* make sense if the modified files are only rarely modified, unlike index files.

Again, this *could* be implemented with a `--recover` flag, but ideally, it should be the default behaviour.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.