ChainSafe / ChainSafe/lodestar

[fulu] Self rate limiter failed to track completed errored response

Open
#8,256 1 comment 0 reactions 0 assignees View on GitHub
meta-bug spec-fulu :zebra:
Dominant language
TypeScript
Stars
1.4k
Forks
483
Avg merge
1d 16h
Merged PRs (30d)
150

Description

### Describe the bug

this happens as part of #8247 where we write finalizedSync e2e test post-electra
- NodeA (receiver) returned an error
- NodeB (sender) was not able to track the request, specifically in its `finally` block https://github.com/ChainSafe/lodestar/blob/9ed83170648e6d1bf2935b33b7cef2e04519303f/packages/reqresp/src/ReqResp.ts#L212
- although the error was thrown inside https://github.com/ChainSafe/lodestar/blob/9ed83170648e6d1bf2935b33b7cef2e04519303f/packages/reqresp/src/request/index.ts#L214

```
grep -e "requestId=9" -rn 3.txt
3.txt:3860:Eph 4/1 0.156[FinalizedSync-Node-B/network] debug: @@@ Req sending peer=16Uiu2HAmPFy4VFfZb4nBbj62tRkwdzJsYcHqiQ1DwsvxGaz3ahw8, protocols=/eth2/beacon_chain/req/data_column_sidecars_by_range/1/ssz_snappy, requestId=9
3.txt:3861:Eph 4/1 0.156[FinalizedSync-Node-B/network] debug: Req dialing peer method=data_column_sidecars_by_range, version=1, encoding=ssz_snappy, client=Lodestar, peer=16...z3ahw8, requestId=9
3.txt:3895:Eph 4/1 0.159[FinalizedSync-Node-B/network] debug: Req sending request method=data_column_sidecars_by_range, version=1, encoding=ssz_snappy, client=Lodestar, peer=16...z3ahw8, requestId=9
3.txt:3901:Eph 4/1 0.160[FinalizedSync-Node-B/network] debug: Req request sent method=data_column_sidecars_by_range, version=1, encoding=ssz_snappy, client=Lodestar, peer=16...z3ahw8, requestId=9
3.txt:3907:Eph 4/1 0.162[FinalizedSync-Node-A/network] debug: Req received method=data_column_sidecars_by_range, version=1, client=Lodestar, peer=16...vTEixX, requestId=9
3.txt:3913:Eph 4/1 0.164[FinalizedSync-Node-A/network] verbose: Resp error method=data_column_sidecars_by_range, version=1, client=Lodestar, peer=16...vTEixX, requestId=9, status=2, errorMessage=No finalized dataColumnSidecar found for slot=8, index=0
3.txt:3955:Eph 4/1 0.178[FinalizedSync-Node-B/network] verbose: Req error method=data_column_sidecars_by_range, version=1, encoding=ssz_snappy, client=Lodestar, peer=16...z3ahw8, requestId=9, status=2, errorMessage=RESPONSE_ERROR_SERVER_ERROR: No finalized dataColumnSidecar found for slot=8, index=0
3.txt:3966:Eph 4/1 0.180[FinalizedSync-Node-B/network] debug: Req error instance of ResponseError method=data_column_sidecars_by_range, version=1, encoding=ssz_snappy, client=Lodestar, peer=16...z3ahw8, requestId=9, status=2, errorMessage=RESPONSE_ERROR_SERVER_ERROR: No finalized dataColumnSidecar found for slot=8, index=0
```

- as a result, subsequent requests failed with
```
Eph 4/1 0.193[FinalizedSync-Node-B/sync] verbose: Batch download error id=Finalized-0, startEpoch=2, status=Downloading, peer=16...z3ahw8, code=REQUEST_ERROR_SELF_RATE_LIMITED
Error: REQUEST_ERROR_SELF_RATE_LIMITED
at ReqRespBeaconNode.sendRequest (/Users/tuyennguyen/Projects/lodestar/packages/reqresp/src/ReqResp.ts:179:15)
at sendRequest.next ()
at collectSequentialBlocksInRange (/Users/tuyennguyen/Projects/lodestar/packages/beacon-node/src/network/reqresp/utils/collectSequentialBlocksInRange.ts:17:20)
at Network.sendBeaconBlocksByRange (/Users/tuyennguyen/Projects/lodestar/packages/beacon-node/src/network/network.ts:514:12)
at beaconBlocksMaybeBlobsByRange (/Users/tuyennguyen/Projects/lodestar/packages/beacon-node/src/network/reqresp/beaconBlocksMaybeBlobsByRange.ts:154:19)
at SyncChain.RangeSync.downloadBeaconBlocksByRange (/Users/tuyennguyen/Projects/lodestar/packages/beacon-node/src/sync/range/range.ts:209:12)
at SyncChain.sendBatch (/Users/tuyennguyen/Projects/lodestar/packages/beacon-node/src/sync/range/chain.ts:454:14)
at SyncChain.requestBatches (/Users/tuyennguyen/Projects/lodestar/packages/beacon-node/src/sync/range/chain.ts:379:19)
at SyncChain.triggerBatchDownloader (/Users/tuyennguyen/Projects/lodestar/packages/beacon-node/src/sync/range/chain.ts:344:12)
at SyncChain.sendBatch (/Users/tuyennguyen/Projects/lodestar/packages/beacon-node/src/sync/range/chain.ts:513:12)
```

- can reproduce in #8255
- log is posted in https://github.com/ChainSafe/lodestar/issues/8247#issuecomment-3213947886

### Expected behavior

- in case of error response, self rate limiter should be able to handle it

### Steps to reproduce

_No response_

### Additional context

_No response_

### Operating system

Linux

### Lodestar version or commit hash

unstable

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.