ChainSafe / ChainSafe/lodestar
[fulu] Self rate limiter failed to track completed errored response
- Dominant language
- TypeScript
- Stars
- 1.4k
- Forks
- 483
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 150
Description
### Describe the bug
this happens as part of #8247 where we write finalizedSync e2e test post-electra
- NodeA (receiver) returned an error
- NodeB (sender) was not able to track the request, specifically in its `finally` block https://github.com/ChainSafe/lodestar/blob/9ed83170648e6d1bf2935b33b7cef2e04519303f/packages/reqresp/src/ReqResp.ts#L212
- although the error was thrown inside https://github.com/ChainSafe/lodestar/blob/9ed83170648e6d1bf2935b33b7cef2e04519303f/packages/reqresp/src/request/index.ts#L214
```
grep -e "requestId=9" -rn 3.txt
3.txt:3860:Eph 4/1 0.156[FinalizedSync-Node-B/network] debug: @@@ Req sending peer=16Uiu2HAmPFy4VFfZb4nBbj62tRkwdzJsYcHqiQ1DwsvxGaz3ahw8, protocols=/eth2/beacon_chain/req/data_column_sidecars_by_range/1/ssz_snappy, requestId=9
3.txt:3861:Eph 4/1 0.156[FinalizedSync-Node-B/network] debug: Req dialing peer method=data_column_sidecars_by_range, version=1, encoding=ssz_snappy, client=Lodestar, peer=16...z3ahw8, requestId=9
3.txt:3895:Eph 4/1 0.159[FinalizedSync-Node-B/network] debug: Req sending request method=data_column_sidecars_by_range, version=1, encoding=ssz_snappy, client=Lodestar, peer=16...z3ahw8, requestId=9
3.txt:3901:Eph 4/1 0.160[FinalizedSync-Node-B/network] debug: Req request sent method=data_column_sidecars_by_range, version=1, encoding=ssz_snappy, client=Lodestar, peer=16...z3ahw8, requestId=9
3.txt:3907:Eph 4/1 0.162[FinalizedSync-Node-A/network] debug: Req received method=data_column_sidecars_by_range, version=1, client=Lodestar, peer=16...vTEixX, requestId=9
3.txt:3913:Eph 4/1 0.164[FinalizedSync-Node-A/network] verbose: Resp error method=data_column_sidecars_by_range, version=1, client=Lodestar, peer=16...vTEixX, requestId=9, status=2, errorMessage=No finalized dataColumnSidecar found for slot=8, index=0
3.txt:3955:Eph 4/1 0.178[FinalizedSync-Node-B/network] verbose: Req error method=data_column_sidecars_by_range, version=1, encoding=ssz_snappy, client=Lodestar, peer=16...z3ahw8, requestId=9, status=2, errorMessage=RESPONSE_ERROR_SERVER_ERROR: No finalized dataColumnSidecar found for slot=8, index=0
3.txt:3966:Eph 4/1 0.180[FinalizedSync-Node-B/network] debug: Req error instance of ResponseError method=data_column_sidecars_by_range, version=1, encoding=ssz_snappy, client=Lodestar, peer=16...z3ahw8, requestId=9, status=2, errorMessage=RESPONSE_ERROR_SERVER_ERROR: No finalized dataColumnSidecar found for slot=8, index=0
```
- as a result, subsequent requests failed with
```
Eph 4/1 0.193[FinalizedSync-Node-B/sync] [36mverbose[39m: Batch download error id=Finalized-0, startEpoch=2, status=Downloading, peer=16...z3ahw8, code=REQUEST_ERROR_SELF_RATE_LIMITED
Error: REQUEST_ERROR_SELF_RATE_LIMITED
at ReqRespBeaconNode.sendRequest (/Users/tuyennguyen/Projects/lodestar/packages/reqresp/src/ReqResp.ts:179:15)
at sendRequest.next ()
at collectSequentialBlocksInRange (/Users/tuyennguyen/Projects/lodestar/packages/beacon-node/src/network/reqresp/utils/collectSequentialBlocksInRange.ts:17:20)
at Network.sendBeaconBlocksByRange (/Users/tuyennguyen/Projects/lodestar/packages/beacon-node/src/network/network.ts:514:12)
at beaconBlocksMaybeBlobsByRange (/Users/tuyennguyen/Projects/lodestar/packages/beacon-node/src/network/reqresp/beaconBlocksMaybeBlobsByRange.ts:154:19)
at SyncChain.RangeSync.downloadBeaconBlocksByRange (/Users/tuyennguyen/Projects/lodestar/packages/beacon-node/src/sync/range/range.ts:209:12)
at SyncChain.sendBatch (/Users/tuyennguyen/Projects/lodestar/packages/beacon-node/src/sync/range/chain.ts:454:14)
at SyncChain.requestBatches (/Users/tuyennguyen/Projects/lodestar/packages/beacon-node/src/sync/range/chain.ts:379:19)
at SyncChain.triggerBatchDownloader (/Users/tuyennguyen/Projects/lodestar/packages/beacon-node/src/sync/range/chain.ts:344:12)
at SyncChain.sendBatch (/Users/tuyennguyen/Projects/lodestar/packages/beacon-node/src/sync/range/chain.ts:513:12)
```
- can reproduce in #8255
- log is posted in https://github.com/ChainSafe/lodestar/issues/8247#issuecomment-3213947886
### Expected behavior
- in case of error response, self rate limiter should be able to handle it
### Steps to reproduce
_No response_
### Additional context
_No response_
### Operating system
Linux
### Lodestar version or commit hash
unstable
Contributor guide
Assessment
This issue has not been assessed yet.