ipfs / ipfs/kubo

Feature: ipfs add/get --reflink

Open
#7,557 2 comments 4 reactions 0 assignees View on GitHub
kind/enhancement
Dominant language
Go
Stars
17.1k
Forks
3.2k
Avg merge
3d 18h
Merged PRs (30d)
11

Description

Although filestores are nice, they carry the issue of someone carelessly changing the file content and the cost of checks related to this problem. With a bit of filesystem magic, we can actually do a basically no-cost copy of data from a file to the blockstore or vice versa. The specific API is FICLONERANGE (Linux) and FSCTL_DUPLICATE_EXTENTS_TO_FILE (Windows).

A new boolean / string option `--reflink` should be added to `ipfs add` and `ipfs get`. It accepts three values: false (disable reflink), auto (enable reflink with fallback), and true (enable reflink no fallback).

- For `ipfs add`, `--reflink` only has an effect when the block store is a `flatfs`. `--nocopy` will disable `--reflink` since the block store isn't even used.
- For `ipfs get`, `--reflink` has an effect in two cases: when the block store is `flatfs` and when a block is being retrived from a `filestore`. The proposed `--nocopy` option (#6687) will work with this switch.

A big issue here is the datastore interface, as many bunches of `file:offset:length` information will need to be passed around for reflinking to happen in flatfs. (It *could* happen with other stores but no serious concurrent backend would expose these sort of internal data structure.) It might be easier for filestore since it already does that.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.