celestiaorg / celestiaorg/go-header

bug: sync stuck due to networking error

Open
#240 5 comments 0 reactions 1 assignee Claimed by @vgonkivs View on GitHub
Dominant language
Go
Stars
21
Forks
33
Avg merge
5d 20h
Merged PRs (30d)
7

Description

reproduce on mocha network
1. setup a new full node to sync,
2. close the network(turn off wifi, pull out wire, etc)
3. wait for all peers to fail
4. open the network
5. the sync goroutine will never recover

the sync goroutine is stucking
image

if network is down, the peers in peerQuene will always be in a decreasing state due to errors that not a ErrNotFound one(network connect fail etc). and eventually run out of the havePeer channel. in this time GetRangeByHeight alway wait for hasPeer channel while getRangeByHeight wait for the result channel

a candidate fix is push back the peer state for errEmptyResponse error. https://github.com/celestiaorg/go-header/pull/238

but another fix is to add timeout for ```GetRangeByHeight```
```go
func (s *Syncer[H]) requestHeaders(
ctx context.Context,
fromHead H,
to uint64,
) error {
amount := to - fromHead.Height()
// start requesting headers until amount remaining will be 0
for amount > 0 {
size := header.MaxRangeRequestSize
if amount < size {
size = amount
}

to := fromHead.Height() + size + 1
s.metrics.rangeRequestStart()
//to fix , add timeout for this context
headers, err := s.getter.GetRangeByHeight(ctx, fromHead, to)
s.metrics.updateGetRangeRequestInfo(s.ctx, int(size)/100, err != nil)
s.metrics.rangeRequestStop()
if err != nil {
return err
}

if err := s.storeHeaders(ctx, headers...); err != nil {
return err
}

amount -= size // size == len(headers)
fromHead = headers[len(headers)-1]
}
return nil
}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.