SkyQuery hangs: concurrency design deficiencies lead to thread pool exhaustion, priority inversion
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
### Description of the problem / feature request:
Queries using [SkyQuery](https://docs.bazel.build/versions/main/query.html#sky-query) may hang the Bazel server due to design deficiencies in the parallel query engine used by SkyQuery. This query engine's threading model uses fixed-size shared thread pools, which can lead to priority inversion. It will also use one blocking task for each function that does parallel graph traversal, which can lead to thread exhaustion. This is referenced as a necessary cleanup [in this TODO](https://github.com/bazelbuild/bazel/blob/09c621e4cf5b968f4c6cdf905ab142d5961f9ddc/src/main/java/com/google/devtools/build/lib/query2/ParallelVisitorUtils.java#L77-L79).
Issue #13359 is an example of this issue, as is [a recent comment on issue #906 (but not issue #906 in general)](https://github.com/bazelbuild/bazel/issues/906#issuecomment-873320707).
### Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
To observe threadpool exhaustion, fork https://github.com/BenHenning/demo-bazel-query-issue and run the sole Github Action workflow.
### What operating system are you running Bazel on?
This impacts all operating systems.
### What's the output of `bazel info release`?
4.1.0
Contributor guide
Assessment
This issue has not been assessed yet.