Add proper limits & backpressure
- Dominant language
- Go
- Stars
- 17.1k
- Forks
- 3.2k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 11
Description
### Checklist
- [X] My issue is specific & actionable.
- [X] I am not suggesting a protocol enhancement.
- [X] I have searched on the [issue tracker](https://github.com/ipfs/kubo/issues?q=is%3Aissue) for my issue.
### Description
Many Kubo components do not define limits nor apply backpressure on callers when the limits are reached. This generally manifests as Kubo producing work faster than some internal resource can process it (such as writing to a datastore, performing connection handshakes, finding block providers, etc.). Instead of returning an error to the caller, Kubo tends to accept the work into some queue, often by spawning unbounded goroutines. This leads to some of the most frustrating behavior of Kubo for users: goroutine pileups, uncontrollable resource utilization, OOMs, etc.
Fundamentally, any work generated by an external request should have a configurable limit. The highest-payoff and easiest-to-implement limits though are at the edges where Kubo accepts external work, such as HTTP servers, stream handlers, etc. These would provide a path forward for users to control resource utilization, regardless of where the specific internal issue is.
There are also places internally that generate highly variable amounts of work depending on the request, such as pinning DAGs, that should have configurable limits (e.g anything involving DAGs could have a limit on the number of processed blocks/bytes). This lets servers protect their availability and resource usage for their preferred workload.
When limits are reached, ideally Kubo would propagate some information back to the caller to help them apply appropriate backpressure. But even just dropping the request on the floor is better than accepting unbounded work (the current behavior). In some cases "backpressure" can be conveyed with existing mechanisms such as a stream reset, assuming a stream reset doesn't already have some other meaning like "error"...clients may want to retry on backpressure but not on errors that will never succeed. There is also a UX concern here to make sure that users have a path forward if they are limited--does the API support getting the data in a way where they aren't throttled?
The libp2p resource manager aims to do this at the libp2p layer, but it's not enough for Kubo, because large amounts of resources are consumed above the libp2p layer, such as in Bitswap and IPLD. We really need application limits here, also because tuning application limits is much more intuitive than low-level libp2p limits.
I think the "done" criteria for this would then be:
- Configurable rate limits on
- DHT Server
- Overall and per-message (FIND_NODE, PING, GET_VALUE, PUT_VALUE, ADD_PROVIDER, GET_PROVIDERS)
- Bitswap
- We should focus on non-response messages that are not the result of requests we made to a peer (those can be controlled by limiting the gateway request rate, which is already user-controlled)
- Limit on want-haves
- Limit on want-blocks
- Rate limits on HTTP Gateway and RPC API
- These are lower priority since they are generally user-controlled and in production environments are easy to control with reverse proxies / load balancers
These should be implemented in a way that results in graceful handling by existing clients. This means e.g. no new protobuf fields. We can add those later but I'd like to focus on the minimal implementation here to protect nodes in the existing network.
Once there are some basic controls for "work accepted" then we can add more fine-tuned limits and knobs for internal work queues.
Contributor guide
Assessment
This issue has not been assessed yet.