ipfs files concat [ <local paths> | <cids> ]
- Dominant language
- Go
- Stars
- 17.1k
- Forks
- 3.2k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 11
Description
> Documenting discussion with @ikreymer, @rangerMauve and @ribasushi
We are missing a high level API for concatenating existing UnixFS files into bigger ones.
Having it would allow for improved deduplication in scenarios when bigger archives in formats like WARC (https://webrecorder.net) consist in big part of smaller files that are already on IPFS, allowing for CID/DAG reuse.
## Use cases
- building big files from preexisting DAGs (e.g. WARC from https://webrecorder.net assembled from standalone files)
- (TBD) if we include dirs, then also
- more friendly replacement for [deprecated](https://github.com/ipfs/go-ipfs/issues/7936) `ipfs object patch append-data`
- merging multiple directories without mutating them in MFS
## Proposed design
Add `concat` command to `ipfs files` that accepts two or more UnixFS-compatible DAGs and returns a CID that is a logical concatenation of all DAGs.
```console
$ ipfs files concat [ /local/mfs/paths | /ipfs/cids ]
bafy....
```
### FAQ / Open questions
We need to agree how to handle edge cases, below are my initial ideas,
feedback on ergonomics and potential implementation caveats is appreciated
- What happens when passed DAGs are all files?
- Concatenate them in-order and produce a new UnixFS file that is reusing original DAGs (maximizing deduplication)
- Should this support directories? It opens additional questions:
- What happens when passed DAGs are all directories?
- Create a new directory which has all children from original directories (in-order?)
- What happens when the first DAG is a directory and all remaining ones are files?
- Create a new directory which has remaining files added
- What happens when the first DAG is a file but at least one of the remaining ones is a directory?
- (A) Return "Error: concatenating directories is possible only when the first DAG is an UnixFS directory"
- (B) Concatenate everything into a single UnixFS directory (children from directories + standalone files)
- What happens if the same CID is in two directories under the same name?
- Should it be duplicated or deduplicated?
Contributor guide
Assessment
This issue has not been assessed yet.