grpc / grpc/grpc-rust

`tonic-health`: add `NOT_SERVING` override for all registered services

Open
#1,693 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
12.5k
Forks
1.3k
Avg merge
4d 7h
Merged PRs (30d)
24

Description

## Feature Request

Add ability to mask all registered services as `NOT_SERVING` despite if it individually has state `SERVING`.

### Crates

`tonic-health`

### Motivation

The [`HealthReporter`](https://github.com/hyperium/tonic/blob/068421a0bdd966eada1cc2aab70fa0845a4bb87a/tonic-health/src/server.rs#L41-L104) implementation does not easily support the use case of "draining a server" to inform a load balancer to stop sending requests to a particular instance of a server during the shutdown grace period, it only has functionality to change the state on one service at a time. For context for anyone not familiar with the issue, here is a troubleshooting guide for [Google's GCP load balancer](https://cloud.google.com/kubernetes-engine/docs/troubleshooting/load-balancing#500-series-errors) for example.

So basically, make it easy to say everything is not serving. For example in Kubernetes, a loadbalancer could route to multiple `Services` for gRPC which could all happen point to the single `tonic` `Deployment` . When the deployment's `Pod` is being restarted all services need to drain at the same time.

### Proposal

Refactor `HealthReporter` so that a not-serving flag is also RW-locked to override the status of any registered service when requested.

Basically usage would be like the following, but I have no strong opinion about the actual function names:

```rust
health_reporter
.override_all_not_serving()
.await;

health_reporter
.clear_override_all_not_serving()
.await;
```

### Alternatives

- maybe write an interceptor for the health check service?
- copy/paste fork the health check server code to add the global flag there

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.