ipfs / ipfs/kubo

GC strategy for public DSNlink gateways

Open
#7,784 0 comments 2 reactions 0 assignees View on GitHub
need/triage topic/docs-ipfs
Dominant language
Go
Stars
17.1k
Forks
3.2k
Avg merge
3d 18h
Merged PRs (30d)
11

Description

This issue is partly a documentation request (because I don't think there are docs that cover a deployment like this) and partly an enhancement request (because I think some new features are needed in go-ipfs to support this) and partly a discussion thread (because maybe the community will have some clever ideas for me :))

I have a bunch of data that I want to host on IPFS, making it available to anyone running an IPFS node, or via a public gateway. The overall size of the content is quite large (several terabytes), but it's not expected to be widely used. So given the relatively low bandwidth requirements, I thought I'd set-up a VPS as a public gateway, using DNSLink so that it'll only serve content my content. But given the large overall size, I don't want to provision the VPS with multi-terabytes of storage, as this his cost prohibitive. Instead I'd like to provision a few hundred gigabytes of local storage as a "cache".

I've set this up, and it's running great. The full content is provided by a handful of nodes in the IPFS network, and this public HTTP gateway with DNSLink keeps the common blocks cached locally for speedy responses. Anything not found locally is pulled from the IPFS network (BTW, being able to have each node maintain a permanent connection to my gateway would be a very useful feature)

My only problem now is how to implement a garbage collection strategy on my public gateway. None of the content is pinned, so running `ipfs repo gc` will wipe everything. I think for my particular use-case, a GC mode that prunes least-recently used blocks would be ideal. Another possible strategy might be to GC the largest files first.

* Is there anything available to me today to help support this use case?
* What do the public gateways at ipfs.io do to keep their repo sizes manageable?
* Are there any clever things I could do by inspecting the repo directly? For example, if I enabled atime on my repo, could I safely `rm` blocks in `$IPFS_PATH/blocks/` that haven't been accessed in a while?

Thanks!

Contributor guide

Open the contributing guide

Research direction

Start with the current `ipfs repo gc` behavior and the `$IPFS_PATH/blocks/` layout described in the issue, then review how public gateways manage repository size. Compare the proposed least-recently-used and largest-file strategies with the existing pinning and cache setup. Done would require a clearly defined, supported strategy or documented guidance for this deployment.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.