Improve performance of Bazel query streamed protobuf output
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
### Description of the feature request:
(description copied from related issue https://github.com/bazelbuild/bazel/issues/24293)
Our repository contains about 700,000 targets. We use the output of `bazel query` to improve CI performance, by restricting the Bazel build to changed targets and their transitive dependencies (similar to [bazel-diff](https://github.com/Tinder/bazel-diff)).
Specifically, we run:
```
bazel query --output=streamed_proto //...
```
This produces a 6.8 GB file and takes (~cold):
- 16 seconds to download/unpack external repos
- 14 seconds to parse all the BUILD.bazel files
- 4 seconds to evaluate the query expression
- 1 minute, 35 seconds to render the protos to stdout
We'd like to speed up this last step, as it’s 74% of wall time.
Through Java profiling (via YourKit and Java Flight Recorder) we've noticed that Bazel spends a lot of wall time on a single thread doing:
1. Constructing protobuf `Build.Target` objects in memory
2. Rendering said proto to varint-delimited wire format
3. Actually writing to `stdout`

### Which category does this issue belong to?
Core, Performance
### What underlying problem are you trying to solve with this feature?
Improve `bazel query --output=streamed_proto` performance
### Which operating system are you running Bazel on?
Linux Ubuntu 24.04.1
### What is the output of `bazel info release`?
release 7.2.0
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
_No response_
### What's the output of `git remote get-url origin; git rev-parse HEAD` ?
_No response_
### Have you found anything relevant by searching the web?
_No response_
### Any other information, logs, or outputs that you want to share?
_No response_
Contributor guide
Research direction
Start with the related issue #24293 and the `bazel query --output=streamed_proto` path, focusing on the profiled work that constructs `Build.Target` objects, renders varint-delimited output, and writes to stdout. Measure the current rendering time on a representative large repository; done means faster streamed protobuf output without changing its contents.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100