linkerd / linkerd/linkerd2

bug(proxy): proxy does not adhere to prescribed resource limits

Open
#15,119 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

pinned
Dominant language
Go
Stars
11.5k
Forks
1.4k
Avg merge
3d 22h
Merged PRs (30d)
53

Description

there is an issue in how we configure our proxy's async runtime. see `linkerd2_proxy::rt::build()`, here:

```rust
match workers.cores().get() {
1 => { /* elided for brevity... */ }
cores => {
info!(%cores, "Using multi-threaded proxy runtime");
Builder::new_multi_thread()
.enable_all()
.thread_name("proxy")
.worker_threads(cores)
.max_blocking_threads(cores)
.build()
.expect("failed to build threaded runtime!")
}
}
```

when told to establish a runtime with N > 1 cores, we use tokio's multi-threaded runtime. we tell it to use N worker threads, with a maximum of N blocking threads.

however, note the documentation of `max_blocking_threads()`:

> Specifies the limit for **additional** threads spawned by the Runtime.
>
> These threads are used for blocking operations like tasks spawned through
> `spawn_blocking`, this includes but is not limited to:
>
> - fs operations
> - dns resolution through `ToSocketAddrs`
> - writing to `Stdout` or `Stderr`
> - reading from `Stdin`
>
> Unlike the `worker_threads`, they are not always active and will exit if
> left idle for too long. You can change this timeout duration with
> `thread_keep_alive`.
>
> **In old versions `max_threads` limited both blocking and worker threads, but
> the current `max_blocking_threads` does not include async worker threads in
> the count.**

\-

emphasis mine.

in other words, this code is not limiting the cores used by linkerd2-proxy to N cores as stated. `max_blocking_threads` is not inclusive of the threads used to service asynchronous workers, and instead is limiting the number of **additional** workers spawned.

so the actual upper-bound is `N*2` cores used by the runtime.

this manifests in reports like https://github.com/linkerd/linkerd2/issues/14813. while they had configured the proxy to use a maximum of 40 cores, they observed the proxy using 48 cores.

linkerd/linkerd2-proxy#826 upgraded to tokio 1.0, and this breaking change to runtime configuration slipped by during that process. `tokio::Runtime::builder::max_threads` method was not just renamed! this new method limits the runtime across a different dimension.

for reference, some relevant upstream issues connected to this:

* https://github.com/tokio-rs/tokio/issues/2802#issuecomment-763889643
* https://github.com/tokio-rs/tokio/issues/2720

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at linkerd2_proxy::rt::build() and read Tokio's Builder documentation for max_blocking_threads alongside the linked runtime configuration change. Trace how worker and blocking threads are counted, then verify that a runtime configured for N cores does not exceed that prescribed limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
infrastructure, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.