`tonic-health`: add `NOT_SERVING` override for all registered services
- Dominant language
- Rust
- Stars
- 12.5k
- Forks
- 1.3k
- Avg merge
- 4d 7h
- Merged PRs (30d)
- 24
Description
## Feature Request
Add ability to mask all registered services as `NOT_SERVING` despite if it individually has state `SERVING`.
### Crates
`tonic-health`
### Motivation
The [`HealthReporter`](https://github.com/hyperium/tonic/blob/068421a0bdd966eada1cc2aab70fa0845a4bb87a/tonic-health/src/server.rs#L41-L104) implementation does not easily support the use case of "draining a server" to inform a load balancer to stop sending requests to a particular instance of a server during the shutdown grace period, it only has functionality to change the state on one service at a time. For context for anyone not familiar with the issue, here is a troubleshooting guide for [Google's GCP load balancer](https://cloud.google.com/kubernetes-engine/docs/troubleshooting/load-balancing#500-series-errors) for example.
So basically, make it easy to say everything is not serving. For example in Kubernetes, a loadbalancer could route to multiple `Services` for gRPC which could all happen point to the single `tonic` `Deployment` . When the deployment's `Pod` is being restarted all services need to drain at the same time.
### Proposal
Refactor `HealthReporter` so that a not-serving flag is also RW-locked to override the status of any registered service when requested.
Basically usage would be like the following, but I have no strong opinion about the actual function names:
```rust
health_reporter
.override_all_not_serving()
.await;
health_reporter
.clear_override_all_not_serving()
.await;
```
### Alternatives
- maybe write an interceptor for the health check service?
- copy/paste fork the health check server code to add the global flag there
Contributor guide
Assessment
This issue has not been assessed yet.