bug(proxy): proxy does not adhere to prescribed resource limits
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 11.5k
- Forks
- 1.4k
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 53
Description
there is an issue in how we configure our proxy's async runtime. see `linkerd2_proxy::rt::build()`, here:
```rust
match workers.cores().get() {
1 => { /* elided for brevity... */ }
cores => {
info!(%cores, "Using multi-threaded proxy runtime");
Builder::new_multi_thread()
.enable_all()
.thread_name("proxy")
.worker_threads(cores)
.max_blocking_threads(cores)
.build()
.expect("failed to build threaded runtime!")
}
}
```
when told to establish a runtime with N > 1 cores, we use tokio's multi-threaded runtime. we tell it to use N worker threads, with a maximum of N blocking threads.
however, note the documentation of `max_blocking_threads()`:
> Specifies the limit for **additional** threads spawned by the Runtime.
>
> These threads are used for blocking operations like tasks spawned through
> `spawn_blocking`, this includes but is not limited to:
>
> - fs operations
> - dns resolution through `ToSocketAddrs`
> - writing to `Stdout` or `Stderr`
> - reading from `Stdin`
>
> Unlike the `worker_threads`, they are not always active and will exit if
> left idle for too long. You can change this timeout duration with
> `thread_keep_alive`.
>
> **In old versions `max_threads` limited both blocking and worker threads, but
> the current `max_blocking_threads` does not include async worker threads in
> the count.**
\-
emphasis mine.
in other words, this code is not limiting the cores used by linkerd2-proxy to N cores as stated. `max_blocking_threads` is not inclusive of the threads used to service asynchronous workers, and instead is limiting the number of **additional** workers spawned.
so the actual upper-bound is `N*2` cores used by the runtime.
this manifests in reports like https://github.com/linkerd/linkerd2/issues/14813. while they had configured the proxy to use a maximum of 40 cores, they observed the proxy using 48 cores.
linkerd/linkerd2-proxy#826 upgraded to tokio 1.0, and this breaking change to runtime configuration slipped by during that process. `tokio::Runtime::builder::max_threads` method was not just renamed! this new method limits the runtime across a different dimension.
for reference, some relevant upstream issues connected to this:
* https://github.com/tokio-rs/tokio/issues/2802#issuecomment-763889643
* https://github.com/tokio-rs/tokio/issues/2720
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at linkerd2_proxy::rt::build() and read Tokio's Builder documentation for max_blocking_threads alongside the linked runtime configuration change. Trace how worker and blocking threads are counted, then verify that a runtime configured for N cores does not exceed that prescribed limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- infrastructure, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100