bazelbuild / bazelbuild/bazel

Memory usage of CoverageOutputGenerator is unreasonable

Open
#26,383 3 comments 0 reactions 1 assignee Claimed by @c-mita View on GitHub
coverage P2 type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 18h
Merged PRs (30d)
75

Description

When working on large lcov files, the heap requirements for CoverageOutputGenerator balloon to ridiculous levels.

Given two LCOV files of several hundred megabytes, CoverageOutputGenerator requires a maximum heap size exceeding 10 GB. Even a single lcov file has absurdly large requirements.

```
$ du -h /tmp/lcov/*
745M /tmp/lcov/lcov0.dat

# build an execuabtle version of CoverageOutputGenerator
# Run it with heap size capped to 5 GB
$ cov_gen --jvm_flags='-Xmx5G' --coverage_dir=/tmp/lcov --output_file=/tmp/merged.dat
SEVERE: Unhandled exception on lcov tool: java.lang.OutOfMemoryError
```

There are a couple of issues:

* The peak working set is quite high. Perhaps ~10x the size of the merged report.
* There is potentially very high GC pressure; even for smaller LCOV files (assuming many are being merged), a large number of "temporary" objects (`LineCoverage`, etc) are created and disposed of.

These largely have the same root cause: the internal coverage representation is inefficient; lots of small objects holding redundant data.

An incremental improvement would be to reduce the size of these objects:

* Both `LineCoverage` and `BranchCoverage` hold a "line number" field, but this is largely redundant since it's already known to `SourceFileCoverage` which has to map line number to coverage data already).
* `LineCoverage` could probably be replaced with `Integer`; at least then the Integer cache might provide some savings.

A larger improvement would be to dramatically change how `SourceFileCoverage` is implemented; e.g. replacing the map of line_number->execution count with an array.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.