ipfs-shipyard / ipfs-shipyard/integration-mini-projects

Represent hypercore in IPLD

Open
#15 0 comments 4 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
14
Forks
1
PR merge metrics
No merged PRs in 30d

Description

6 months ago, I did some initial experimentation to see if I could model the [hypercore](https://github.com/mafintosh/hypercore) data structure in IPLD. Hypercore is the append-only log library used by Dat.

https://github.com/jimpick/hypercore-simple-ipld

As hypercore data is stored in [simple binary tree](https://datprotocol.github.io/how-dat-works/), it is very easy to model with IPLD. I didn't get as far as storing signatures, or the bitfield used by the replication protocol, but those shouldn't be hard to model.

I was experimenting with implementing the [random-access-storage](https://github.com/random-access-storage/random-access-storage) interface, so, if completed, it would be possible to read/write hypercore data from an IPFS node directly (similar to how hypercore can use disk storage, or memory). It would be interesting to see what the performance profile is like ... I suspect it should be similar to MFS.

If the storage interface is implemented, it should also be possible to run [hyperdrive](https://github.com/mafintosh/hyperdrive), [hypertrie](https://github.com/mafintosh/hypertrie) and [hyperdb](https://github.com/mafintosh/hyperdb) on top of that. [hypercored](https://github.com/mafintosh/hypercored) could write directly to IPFS or ipfs-cluster.

Because IPFS will deduplicate blocks, this may be a cost saving to some people who are trying to store large amounts of data using Dat. Additionally, ipfs-cluster may be a nice way to manage large amounts of Dat data. And Filecoin could be used to keep archives alive for long periods of time.

It would also be interesting to try to run hypercore replication over libp2p, and to try different approaches for content discovery, but that would be a separate project.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.