grpc / grpc/grpc-rust

Unexpected blocking IO when configuring TLS with tls-native-roots enabled

Open
#2,424 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
12.5k
Forks
1.3k
Avg merge
4d 7h
Merged PRs (30d)
24

Description

## Bug Report

### Version

0.12.3 (feature `tls-native-roots` enabled)

### Platform

Linux

### Description

Hi! I was investigating tokio scheduling lags in a service I work on, and ended up noticing that TLS configuration can take several milliseconds (~20ms). This can pile up and lead to tokio stalls of hundreds of milliseconds, blocking all other processing in the meantime. Our process is part of a data intensive distributed system and communicates with hundreds or thousands of nodes at any point, and connections end up being regularly refreshed (for example because of nodes coming and going away).
Another aggravating factor is that we use a custom load balancer that will configure TLS for new discovered endpoints (and maybe that's an anti-pattern, happy to be guided there).

When looking a bit deeper into it, I noticed that when the `tls-native-roots` is enabled, configuring TLS will lead to [calling this function](https://github.com/hyperium/tonic/blob/ff7b54045c943172de4c65cf949011492db185ee/tonic/src/transport/channel/service/tls.rs#L67C1-L68C1), that (if I understood correctly) will read the certificate chain from disk, at least on Linux.

The code we use is basically something like:

```rust
endpoint.tls_config(ClientTlsConfig::new().with_enabled_roots())
```

Is this a known issue? If yes, what is usually the best practice?
I imagine it would be fine for us to have a thread refresh the native certificate chain in the background and just use this to initialize TLS, but I am not sure how to proceed (and want to avoid over-engineering if there is a better alternative).

Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.