ethereum / ethereum/execution-apis

`health`: Implement a health check standard

Open
#529 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Io
Stars
1.1k
Forks
530
Avg merge
5d 8h
Merged PRs (30d)
9

Description

We recently [opened a PR](https://github.com/ethereum/go-ethereum/pull/29056) on the Go Ethereum repo to implement a `health` API that will work with cloud/loud balancer provider's health checks to determine if a node is in a suitable state to handle requests from clients.

It would be ideal for consensus on a universal standard for such a service to be acquired prior to implementation. @lightclient recommended we open an issue here to get the ball rolling.

### How Health Checks Work

Health checks across large providers generally work the same way ([GCP](https://cloud.google.com/load-balancing/docs/health-check-concepts#criteria-protocol-http), [Azure](https://learn.microsoft.com/en-us/azure/app-service/monitor-instances-health-check?tabs=dotnet), [AWS](https://aws.amazon.com/builders-library/implementing-health-checks/?nc1=h_ls)):

- The provider sends a `GET` request to an endpoint running on the service.
- The service responds either status `200` (healthy), `500` (unhealthy) or doesn't respond at all (unhealthy).
- If the service is unhealthy traffic heading to that piece of infrastructure is routed elsewhere.

There is some nuance where a small number of providers accept `2XX`/`5XX` statuses with different actions for different statuses, however the `200`/`500` model is universal.

### Limitations

Health checks only allow simple `GET` requests. This is at odds with the current Ethereum standard which only allows `POST` requests of type `application/json`.

Because only `GET` requests are allowed you cannot post the parameters in the JSON-RPC 2.0 format. Users must utilise custom headers or query strings in the URL to pass parameters.

### Checks

As a starting point these are the checks we implemented in our PR (derived from [Erigon's solution](https://github.com/ledgerwatch/erigon/blob/devel/cmd/rpcdaemon/README.md#healthcheck)):

- `syncing`: Check if the node is in the syncing state.
- `max_seconds_behind`: Check the timestamp of the latest block isn't greater than `max_seconds_behind` seconds ago.
- `min_peers`: Confirm there are at least `min_peers` number of peers connected to the node.
- `check_block`: Confirm the node has synced beyond the height of `check_block`.

In our implementation we used custom headers to pass the values for each variable. When a variable is not defined that check will not run.

Upon finishing the checks the node returns either status `200` if all checks passed or `500` if any failed. Additionally an object containing the results for each test (`OK`, `DISABLED` or the error message for the check) is returned.

### Further Work

Any assistance on further developing this idea in to a solid standard and getting this moving forward would be hugely appreciated. Such an endpoint would be massively useful to us as node operators and likely many others. Our team has the resources and willingness to assist however possible.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.