grpc / grpc/grpc-rust

unexpected tonic_health error: `failed serving connection: connection error`

Open
#2,181 2 comments 2 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
12.5k
Forks
1.3k
Avg merge
4d 7h
Merged PRs (30d)
24

Description

## Bug Report

### Version

```
> cargo tree | grep tonic
│ │ │ │ │ └── tonic v0.12.3
│ │ │ │ ├── tonic v0.12.3 (*)
│ │ │ └── tonic v0.12.3 (*)
│ │ └── tonic v0.12.3 (*)
│ │ └── tonic v0.12.3 (*)
│ │ └── tonic v0.12.3 (*)
│ │ └── tonic v0.12.3 (*)
│ │ └── tonic v0.12.3 (*)
│ │ └── tonic v0.12.3 (*)
│ │ └── tonic v0.12.3 (*)
│ │ └── tonic v0.12.3 (*)
│ ├── tonic v0.12.3 (*)
│ ├── tonic-health v0.12.3
│ │ └── tonic v0.12.3 (*)
│ └── tonic v0.12.3 (*)
├── tonic v0.12.3 (*)
├── tonic-health v0.12.3 (*)
```

### Platform

```
Linux #16~22.04.1-Ubuntu SMP Mon Aug 19 19:38:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
```

### Crates

```
tonic-health
```

### Description

We're using tonic-health for native kubernetes gRPC health checks (as described [here](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe)).

Here, the relevant snippet from the deployment descriptor:

Image

On each invocation of the health endpoint in our application, we see an error being logged on `DEBUG` level:

```
source: /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tonic-0.12.3/src/transport/server/mod.rs
line: 703
message: failed serving connection: connection error
```

The health check itself is working as expected. We can confirm that the endpoint responds with `SERVING` / `NOT_SERVING` as expected. Therefore, I suspect that the problem may be related to how the connection is terminated.

I would expect that no errors are being logged, if the health-check is working properly, not even on `DEBUG` level, especially with our standard kubernetes setup.

### more info

Relevant initialisation code in main.rs:

```
let (mut health_reporter, health_server) = tonic_health::server::health_reporter();
health_reporter
.set_service_status("", ServingStatus::NotServing)
.await;
let addr = "0.0.0.0:9000".parse().unwrap();
let grpc_server = Server::builder()
.add_service(health_server)
.add_service(
CustomApiServer::new(business_logic_service)
)
.serve_with_shutdown(addr, cancellation_token.clone().cancelled_owned())
.unwrap();
let grpc_server_handle = tokio::spawn(grpc_server);
// set serving happens down the line
```

Kubernetes version: `Server Version: v1.30.5`

We were wondering if [this](https://github.com/hyperium/tonic/issues/485) issue describes our problem, because k8s has a Go-client. But we do specify the host using `0.0.0.0`, so it seems that cannot be the cause.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.