ethereum-optimism / ethereum-optimism/optimism
[oncall] CI: Failing RPC calls on superchain-ops repo [block is out of range]
- Dominant language
- Go
- Stars
- 6.5k
- Forks
- 4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 145
Description
**Description**
The `forge_test` CI workflow for `superchain-ops` repo is failing when setting up fork testing.
[Example failed workflow](https://app.circleci.com/jobs/github/ethereum-optimism/superchain-ops/153096)
[Configuration of `proxyd` it is connecting to](https://github.com/ethereum-optimism/k8s/blob/master/kustomize/proxyd/oplabs-prod-sepolia/proxyd-l2-ci/proxyd.toml)
From the logs below we can see that errors are coming from `proxyd-l2-ci` - [the `proxyd` that the first `proxyd` is calling](https://github.com/ethereum-optimism/k8s/blob/master/kustomize/proxyd/oplabs-prod-sepolia/proxyd-l2-consensus/proxyd.toml#L4)
From there it is one of these:
```
[backends.replica-0]
consensus_skip_peer_count = true
rpc_url = "https://op-opn-geth-a-rpc-0-op-geth.secondary.sepolia.prod.oplabs.cloud"
ws_url = "ws://dummy"
[backends.replica-1]
consensus_skip_peer_count = true
rpc_url = "https://op-opn-geth-a-rpc-1-op-geth.secondary.sepolia.prod.oplabs.cloud"
ws_url = "ws://dummy"
[backends.replica-2]
consensus_skip_peer_count = true
rpc_url = "https://op-opn-geth-a-rpc-2-op-geth.secondary.sepolia.prod.oplabs.cloud"
ws_url = "ws://dummy"
```
[Logs from the `proxyd` it is connecting to](https://argocd.primary.infra.tools.oplabs.cloud/applications/prod-sepolia-proxyd-l2-ci?resource=&node=%2FPod%2Fproxyd-l2-ci%2Fproxyd-5db99d5cc9-5lgcf%2F0&tab=logs):
```
{"time":"2025-09-19T18:12:03.131512791Z","level":"WARN","msg":"backend request failed, trying again","name":"proxyd-l2-ci","req_id":"6d27217d4b293ad8c679","err":"response code 503","method":"eth_getBlockByHash","attempt_count":1,"max_attempts":1}
{"time":"2025-09-19T18:12:03.131606421Z","level":"ERROR","msg":"error forwarding request to backend","name":"proxyd-l2-ci","req_id":"6d27217d4b293ad8c679","auth":"none","err":"permanent error forwarding request response code 503"}
{"time":"2025-09-19T18:12:03.131641971Z","level":"ERROR","msg":"error serving requests","req_id":"6d27217d4b293ad8c679","auth":"none","err":"no backend is currently healthy to serve traffic"}
{"time":"2025-09-19T18:12:03.131659931Z","level":"ERROR","msg":"error forwarding RPC batch","batch_size":1,"backend_group":"proxyd-l2-ci","req_id":"6d27217d4b293ad8c679","err":"no backend is currently healthy to serve traffic"}
{"time":"2025-09-19T18:12:03.268365621Z","level":"WARN","msg":"backend request failed, trying again","name":"proxyd-l2-ci","req_id":"6ba9478b2eacd3bad9c3","err":"response code 503","method":"eth_getBlockByHash","attempt_count":1,"max_attempts":1}
{"time":"2025-09-19T18:12:03.268413511Z","level":"ERROR","msg":"error forwarding request to backend","name":"proxyd-l2-ci","req_id":"6ba9478b2eacd3bad9c3","auth":"none","err":"permanent error forwarding request response code 503"}
{"time":"2025-09-19T18:12:03.268452921Z","level":"ERROR","msg":"error serving requests","req_id":"6ba9478b2eacd3bad9c3","auth":"none","err":"no backend is currently healthy to serve traffic"}
{"time":"2025-09-19T18:12:03.268498941Z","level":"ERROR","msg":"error forwarding RPC batch","batch_size":1,"backend_group":"proxyd-l2-ci","req_id":"6ba9478b2eacd3bad9c3","err":"no backend is currently healthy to serve traffic"}
{"time":"2025-09-19T18:12:03.292863891Z","level":"WARN","msg":"backend request failed, trying again","name":"proxyd-l2-ci","req_id":"c79abefbd2f7fa8682e2","err":"response code 503","method":"eth_getBlockByNumber","attempt_count":1,"max_attempts":1}
{"time":"2025-09-19T18:12:03.292948151Z","level":"ERROR","msg":"error forwarding request to backend","name":"proxyd-l2-ci","req_id":"c79abefbd2f7fa8682e2","auth":"none","err":"permanent error forwarding request response code 503"}
{"time":"2025-09-19T18:12:03.292982611Z","level":"ERROR","msg":"error serving requests","req_id":"c79abefbd2f7fa8682e2","auth":"none","err":"no backend is currently healthy to serve traffic"}
{"time":"2025-09-19T18:12:03.293044511Z","level":"ERROR","msg":"error forwarding RPC batch","batch_size":1,"backend_group":"proxyd-l2-ci","req_id":"c79abefbd2f7fa8682e2","err":"no backend is currently healthy to serve traffic"}
{"time":"2025-09-22T19:27:54.067096661Z","level":"WARN","msg":"backend request failed, trying again","name":"proxyd-l2-ci","req_id":"149aff962a882f8c0d90","err":"error in backend request Post \"http://proxyd.proxyd-l2-consensus.svc.cluster.local:8080\": context canceled","method":"eth_getBlockByHash","attempt_count":1,"max_attempts":1}
{"time":"2025-09-22T19:27:54.067165851Z","level":"ERROR","msg":"error forwarding request to backend","name":"proxyd-l2-ci","req_id":"149aff962a882f8c0d90","auth":"none","err":"permanent error forwarding request error in backend request Post \"http://proxyd.proxyd-l2-consensus.svc.cluster.local:8080\": context canceled"}
{"time":"2025-09-22T19:27:54.067233831Z","level":"ERROR","msg":"error serving requests","req_id":"149aff962a882f8c0d90","auth":"none","err":"no backend is currently healthy to serve traffic"}
{"time":"2025-09-22T19:27:54.067275061Z","level":"ERROR","msg":"error forwarding RPC batch","batch_size":1,"backend_group":"proxyd-l2-ci","req_id":"149aff962a882f8c0d90","err":"no backend is currently healthy to serve traffic"}
{"time":"2025-09-23T10:00:52.364020672Z","level":"WARN","msg":"backend request failed, trying again","name":"proxyd-l2-ci","req_id":"29c3f7bae793a62d685e","err":"error reading response body context canceled","method":"eth_getBlockByHash","attempt_count":1,"max_attempts":1}
{"time":"2025-09-23T10:00:52.364088372Z","level":"ERROR","msg":"error forwarding request to backend","name":"proxyd-l2-ci","req_id":"29c3f7bae793a62d685e","auth":"none","err":"permanent error forwarding request error reading response body context canceled"}
{"time":"2025-09-23T10:00:52.364106242Z","level":"ERROR","msg":"error serving requests","req_id":"29c3f7bae793a62d685e","auth":"none","err":"no backend is currently healthy to serve traffic"}
{"time":"2025-09-23T10:00:52.364359232Z","level":"ERROR","msg":"error forwarding RPC batch","batch_size":1,"backend_group":"proxyd-l2-ci","req_id":"29c3f7bae793a62d685e","err":"no backend is currently healthy to serve traffic"}
{"time":"2025-09-24T04:27:17.841760386Z","level":"INFO","msg":"backend responded with RPC error","backend":"proxyd-l2-ci","last_error_code":-32019,"last_error_msg":"block is out of range","req_id":"438711ea6c96fabdd0ff","source":"rpc","auth":"none"}
{"time":"2025-09-24T10:30:25.516971007Z","level":"WARN","msg":"backend request failed, trying again","name":"proxyd-l2-ci","req_id":"e2a84f1ac4a11ff7adb4","err":"error in backend request Post \"http://proxyd.proxyd-l2-consensus.svc.cluster.local:8080\": context canceled","method":"eth_getBlockByHash","attempt_count":1,"max_attempts":1}
{"time":"2025-09-24T10:30:25.517048287Z","level":"ERROR","msg":"error forwarding request to backend","name":"proxyd-l2-ci","req_id":"e2a84f1ac4a11ff7adb4","auth":"none","err":"permanent error forwarding request error in backend request Post \"http://proxyd.proxyd-l2-consensus.svc.cluster.local:8080\": context canceled"}
{"time":"2025-09-24T10:30:25.517071976Z","level":"ERROR","msg":"error serving requests","req_id":"e2a84f1ac4a11ff7adb4","auth":"none","err":"no backend is currently healthy to serve traffic"}
{"time":"2025-09-24T10:30:25.517115436Z","level":"ERROR","msg":"error forwarding RPC batch","batch_size":1,"backend_group":"proxyd-l2-ci","req_id":"e2a84f1ac4a11ff7adb4","err":"no backend is currently healthy to serve traffic"}
{"time":"2025-09-24T13:38:41.780332298Z","level":"INFO","msg":"backend responded with RPC error","backend":"proxyd-l2-ci","last_error_code":-32019,"last_error_msg":"block is out of range","req_id":"74fb9d92d35e71c78cc1","source":"rpc","auth":"none"}
{"time":"2025-09-24T14:10:07.492847391Z","level":"INFO","msg":"backend responded with RPC error","backend":"proxyd-l2-ci","last_error_code":-32019,"last_error_msg":"block is out of range","req_id":"41a24a4319a464a1bf41","source":"rpc","auth":"none"}
{"time":"2025-09-24T14:42:09.70781107Z","level":"INFO","msg":"backend responded with RPC error","backend":"proxyd-l2-ci","last_error_code":-32019,"last_error_msg":"block is out of range","req_id":"b1c510417b5eeaeb0b0f","source":"rpc","auth":"none"}
{"time":"2025-09-24T18:11:51.89631197Z","level":"INFO","msg":"backend responded with RPC error","backend":"proxyd-l2-ci","last_error_code":-32019,"last_error_msg":"block is out of range","req_id":"a20ceac14cbf760fa072","source":"rpc","auth":"none"}
{"time":"2025-09-24T21:40:38.268995679Z","level":"INFO","msg":"backend responded with RPC error","backend":"proxyd-l2-ci","last_error_code":-32019,"last_error_msg":"block is out of range","req_id":"f48bff1bc0d85f8651ab","source":"rpc","auth":"none"}
{"time":"2025-09-25T17:13:27.544988456Z","level":"INFO","msg":"backend responded with RPC error","backend":"proxyd-l2-ci","last_error_code":-32019,"last_error_msg":"block is out of range","req_id":"581e0f52220f239f89ca","source":"rpc","auth":"none"}
{"time":"2025-09-25T18:37:05.611526475Z","level":"INFO","msg":"backend responded with RPC error","backend":"proxyd-l2-ci","last_error_code":-32019,"last_error_msg":"block is out of range","req_id":"928917b82dba7163e94f","source":"rpc","auth":"none"}
{"time":"2025-09-26T18:15:03.768078525Z","level":"ERROR","msg":"error serving requests","req_id":"b539cb575782c15ea5dd","auth":"none","err":"context canceled"}
{"time":"2025-09-26T18:15:03.768209945Z","level":"ERROR","msg":"error forwarding RPC batch","batch_size":1,"backend_group":"proxyd-l2-ci","req_id":"b539cb575782c15ea5dd","err":"context canceled"}
{"time":"2025-09-26T23:38:49.887767017Z","level":"INFO","msg":"backend responded with RPC error","backend":"proxyd-l2-ci","last_error_code":-32019,"last_error_msg":"block is out of range","req_id":"bfb89ac7a588bdb85138","source":"rpc","auth":"none"}
{"time":"2025-09-29T00:28:37.62003332Z","level":"INFO","msg":"backend responded with RPC error","backend":"proxyd-l2-ci","last_error_code":-32019,"last_error_msg":"block is out of range","req_id":"462dd59f3c3bc52764a2","source":"rpc","auth":"none"}
{"time":"2025-09-29T18:34:53.65570574Z","level":"INFO","msg":"backend responded with RPC error","backend":"proxyd-l2-ci","last_error_code":-32019,"last_error_msg":"block is out of range","req_id":"0a7f2892d1ce2ecc42c7","source":"rpc","auth":"none"}
{"time":"2025-09-29T18:34:53.65671587Z","level":"INFO","msg":"backend responded with RPC error","backend":"proxyd-l2-ci","last_error_code":-32019,"last_error_msg":"block is out of range","req_id":"b95d3f7903bf521900d8","source":"rpc","auth":"none"}
{"time":"2025-09-29T18:34:53.66269133Z","level":"INFO","msg":"backend responded with RPC error","backend":"proxyd-l2-ci","last_error_code":-32019,"last_error_msg":"block is out of range","req_id":"6c28037baf38129f8959","source":"rpc","auth":"none"}
{"time":"2025-09-29T20:27:11.774584579Z","level":"INFO","msg":"backend responded with RPC error","backend":"proxyd-l2-ci","last_error_code":-32019,"last_error_msg":"block is out of range","req_id":"2ecda2697d670faaaf0f","source":"rpc","auth":"none"}
```
[Grafana shows the in for these errors for this `proxyd` instance](https://optimistic.grafana.net/d/TRHRXiBV6/proxyd-k8s?orgId=1&from=2025-09-25T04:00:00.000Z&to=2025-09-25T04:59:59.000Z&timezone=browser&var-instance=prod-sepolia-proxyd-l2-ci&var-backend_name=$__all&var-method=$__all&refresh=1m)
[Grafana shows no data for the replicas for this time range](https://optimistic.grafana.net/d/n-MDITAVz/replica-healthcheck?orgId=1&from=2025-09-25T04:00:00.000Z&to=2025-09-25T20:59:59.000Z&timezone=browser&var-network=sepolia-prod&var-instance=prod-sepolia-op-opn-geth-a-rpc-0&var-instance=prod-sepolia-op-opn-geth-a-rpc-1&var-instance=prod-sepolia-op-opn-geth-a-rpc-2&var-target=$__all&refresh=1m)
Contributor guide
Assessment
This issue has not been assessed yet.