libp2p / libp2p/go-libp2p

Max outgoing connection limit with pooling

Open
#3,370 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/enhancement
Dominant language
Go
Stars
6.9k
Forks
1.3k
Avg merge
13d 21h
Merged PRs (30d)
1

Description

Problem:

* There is no way of limiting connections without killing them (resource manager will make them fail if above threshold)
* To applications above, this looks as if the remote peer is not dialable at all, but it is more of a local resource problem.
* In order to work with shitty routers and ISPs, we need to reduce the total amount of active connections, but without making peers undialable in the process. The solution is to place connections in a waiting pool. Applications can have more fine-tuned control on how long they want to wait for a dial to happen before giving up on it, rather than receiving an error immediately.

Solution:

* Add outgoing-connection pooling. Dials can happen normally while the total number of connections is below a threshold. When the threshold is crossed, dials will wait until existing connections are closed and leave space for new ones.

Example:
* For example, if a router only supports 1100 concurrent connections, and we set the threshold to 1000, and we need to perform a network crawl that requires a total of 7000 connections, we can ensure that we never have more than 1000 outgoing connections without breaking the network crawl (if we were to use resource manager limits) nor the router. This problem exists now with the accelerated DHT client.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the outgoing dial path, resource manager connection limits, and the accelerated DHT client mentioned in the issue. Determine where dials can wait for capacity and how callers should control their wait duration. Done means outgoing connections remain at or below the configured threshold while queued dials proceed when existing connections close.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.