ipfs / ipfs/boxo

ProvideMany: high memory usage when providing tens of millions of CIDs

Open
#354 7 comments 1 reaction 0 assignees View on GitHub
need/triage
Dominant language
Go
Stars
316
Forks
163
Avg merge
4d 12h
Merged PRs (30d)
8

Description

When using BatchProviding, we are not really batching, but sending all the CIDs at the same time to the Router implementing ProvideMany.

To avoid collateral problems, we should actually batch the calls to ProvideMany.

This will help with Reframe Router implementation (https://github.com/ipfs/go-delegated-routing) to avoid huge JSON payloads sent to the server.

We need to find good defaults to still keep FullRT DHT implementation with good performance numbers.

That is like 1/10th of the memory spike observed, we are still searching for other possible problems.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.