bazelbuild / bazelbuild/bazel

C++ coverage collection is way too slow

Open
#26,226 6 comments 0 reactions 0 assignees View on GitHub
coverage P3 stale team-Rules-CPP type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 18h
Merged PRs (30d)
75

Description

### Description of the bug:

We have a test that runs in 25 seconds with coverage enabled, but generating coverage information for it takes over 5 minutes and causes our build actions to fail.

I've traced the problem down to this super-inefficient loop: https://cs.opensource.google/bazel/bazel/+/master:tools/test/collect_cc_coverage.sh;l=121;drc=11717a16b8ecf4e1e601c39c8b4cfe4728954634 . We have almost 3k gcba files in the profile directory. The code is invoking `gcov` and other tools unnecessarily on every file (e.g. running `gcov` over and over again to query its version, or doing "expensive" `basename` and `dirname` calls). And to make it worse, the whole process is sequential when in reality it's eminently parallelizable.

I hacked my way to prototype a fix for these issues by factoring common code out and parallelizing the script with `xargs`, and I could make instrumentation for our problematic test fit under 5 minutes on a 2-core machine. And when I bumped the action to run on a worker with more cores, I saw the runtime drop to 3 minutes.

I'm hesitant to share my changes to the shell script because they were hacky, and I think we could do a _much_ better job by _not_ using shell for this. The whole thing feels very fragile as it is now.

### Which category does this issue belong to?

C++ Rules

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Look at the code. Realize it just doesn't scale.

### Which operating system are you running Bazel on?

Linux, but doesn't matter.

### What is the output of `bazel info release`?

7.6.1

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse HEAD` ?

```text

```

### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

_No response_

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Research direction

The issue points to tools/test/collect_cc_coverage.sh, especially the loop around line 121; start there and trace how gcov and related tools are invoked for profile files. Compare behavior and runtime on a large profile directory, and treat preserving coverage output while avoiding repeated work and scaling beyond sequential processing as done.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, shell
Domain
build-system, performance, testing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.