bazelbuild / bazelbuild/bazel

SkyQuery hangs: concurrency design deficiencies lead to thread pool exhaustion, priority inversion

Open
#13,682 5 comments 0 reactions 1 assignee Claimed by @michaeledgar View on GitHub
not stale P3 query bugs team-Core type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the problem / feature request:

Queries using [SkyQuery](https://docs.bazel.build/versions/main/query.html#sky-query) may hang the Bazel server due to design deficiencies in the parallel query engine used by SkyQuery. This query engine's threading model uses fixed-size shared thread pools, which can lead to priority inversion. It will also use one blocking task for each function that does parallel graph traversal, which can lead to thread exhaustion. This is referenced as a necessary cleanup [in this TODO](https://github.com/bazelbuild/bazel/blob/09c621e4cf5b968f4c6cdf905ab142d5961f9ddc/src/main/java/com/google/devtools/build/lib/query2/ParallelVisitorUtils.java#L77-L79).

Issue #13359 is an example of this issue, as is [a recent comment on issue #906 (but not issue #906 in general)](https://github.com/bazelbuild/bazel/issues/906#issuecomment-873320707).

### Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

To observe threadpool exhaustion, fork https://github.com/BenHenning/demo-bazel-query-issue and run the sole Github Action workflow.

### What operating system are you running Bazel on?

This impacts all operating systems.

### What's the output of `bazel info release`?

4.1.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.