C++ coverage collection is way too slow
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 75
Description
### Description of the bug:
We have a test that runs in 25 seconds with coverage enabled, but generating coverage information for it takes over 5 minutes and causes our build actions to fail.
I've traced the problem down to this super-inefficient loop: https://cs.opensource.google/bazel/bazel/+/master:tools/test/collect_cc_coverage.sh;l=121;drc=11717a16b8ecf4e1e601c39c8b4cfe4728954634 . We have almost 3k gcba files in the profile directory. The code is invoking `gcov` and other tools unnecessarily on every file (e.g. running `gcov` over and over again to query its version, or doing "expensive" `basename` and `dirname` calls). And to make it worse, the whole process is sequential when in reality it's eminently parallelizable.
I hacked my way to prototype a fix for these issues by factoring common code out and parallelizing the script with `xargs`, and I could make instrumentation for our problematic test fit under 5 minutes on a 2-core machine. And when I bumped the action to run on a worker with more cores, I saw the runtime drop to 3 minutes.
I'm hesitant to share my changes to the shell script because they were hacky, and I think we could do a _much_ better job by _not_ using shell for this. The whole thing feels very fragile as it is now.
### Which category does this issue belong to?
C++ Rules
### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
Look at the code. Realize it just doesn't scale.
### Which operating system are you running Bazel on?
Linux, but doesn't matter.
### What is the output of `bazel info release`?
7.6.1
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
_No response_
### What's the output of `git remote get-url origin; git rev-parse HEAD` ?
```text
```
### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.
_No response_
### Have you found anything relevant by searching the web?
_No response_
### Any other information, logs, or outputs that you want to share?
_No response_
Contributor guide
Research direction
The issue points to tools/test/collect_cc_coverage.sh, especially the loop around line 121; start there and trace how gcov and related tools are invoked for profile files. Compare behavior and runtime on a large profile directory, and treat preserving coverage output while avoiding repeated work and scaling beyond sequential processing as done.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, shell
- Domain
- build-system, performance, testing
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100