bazelbuild / bazelbuild/bazel

Improve performance of Bazel query streamed protobuf output

Open
#24,304 1 comment 0 reactions 0 assignees View on GitHub
P2 team-Core team-Performance type: feature request
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the feature request:

(description copied from related issue https://github.com/bazelbuild/bazel/issues/24293)

Our repository contains about 700,000 targets. We use the output of `bazel query` to improve CI performance, by restricting the Bazel build to changed targets and their transitive dependencies (similar to [bazel-diff](https://github.com/Tinder/bazel-diff)).

Specifically, we run:

```
bazel query --output=streamed_proto //...
```

This produces a 6.8 GB file and takes (~cold):
- 16 seconds to download/unpack external repos
- 14 seconds to parse all the BUILD.bazel files
- 4 seconds to evaluate the query expression
- 1 minute, 35 seconds to render the protos to stdout

We'd like to speed up this last step, as it’s 74% of wall time.

Through Java profiling (via YourKit and Java Flight Recorder) we've noticed that Bazel spends a lot of wall time on a single thread doing:
1. Constructing protobuf `Build.Target` objects in memory
2. Rendering said proto to varint-delimited wire format
3. Actually writing to `stdout`

![Screenshot 2024-11-11 at 5 52 59 PM](https://github.com/user-attachments/assets/2918e728-6cd5-451f-b9c6-5960778a0926)

### Which category does this issue belong to?

Core, Performance

### What underlying problem are you trying to solve with this feature?

Improve `bazel query --output=streamed_proto` performance

### Which operating system are you running Bazel on?

Linux Ubuntu 24.04.1

### What is the output of `bazel info release`?

release 7.2.0

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse HEAD` ?

_No response_

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the related issue #24293 and the `bazel query --output=streamed_proto` path, focusing on the profiled work that constructs `Build.Target` objects, renders varint-delimited output, and writes to stdout. Measure the current rendering time on a representative large repository; done means faster streamed protobuf output without changing its contents.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.