erigontech / erigontech/erigon

Caplin: to set boundaries - no how much goroutines can be spawned by for loop

Open
#12,510 3 comments 1 reaction 0 assignees View on GitHub
Caplin
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
465

Description

Found many patterns like:
```
wg := WaitGroup{}
for {
wg.Add(1)
go func() {
defer wg.Done(1)
}
}
wg.Wait()
```

Examples:
`getDutiesProposer`
`connectWithAllPeers`
`batchVerifyAttestations`
`VerifyAgainstIdentifiersAndInsertIntoTheBlobStore`

This pattern spawns unlimited amount of goroutines. All "unlimited" things are bad - because:
- because we have limited cpus/ram/disk/etc...
- even if today some loop doesn't spawn much goroutines - future PR's or future bugs - may change it and we will not notice
- some "for loops" are run in-parallel - means it's will be hard for us to reproduce "worst case scenario" - where N loops spawning much goroutines in same time.

------
let's replace it by next pattern:
```
g := &errgroup.Group{}
g.SetLimit(runtime.GOMAXPROCS(-1))
for {
g.Go(func() error {
})
}
if err := g.Wait(); err != nil {
return err
})
```

Can set any limit you like.

Contributor guide

Open the contributing guide

Research direction

Start by locating the named entry points: getDutiesProposer, connectWithAllPeers, batchVerifyAttestations, and VerifyAgainstIdentifiersAndInsertIntoTheBlobStore. Inspect each loop for unbounded goroutine creation and determine the appropriate bounded-concurrency approach. Done means the identified patterns have explicit limits and their existing completion and error behavior remains validated.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.