Memory usage of CoverageOutputGenerator is unreasonable
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 75
Description
When working on large lcov files, the heap requirements for CoverageOutputGenerator balloon to ridiculous levels.
Given two LCOV files of several hundred megabytes, CoverageOutputGenerator requires a maximum heap size exceeding 10 GB. Even a single lcov file has absurdly large requirements.
```
$ du -h /tmp/lcov/*
745M /tmp/lcov/lcov0.dat
# build an execuabtle version of CoverageOutputGenerator
# Run it with heap size capped to 5 GB
$ cov_gen --jvm_flags='-Xmx5G' --coverage_dir=/tmp/lcov --output_file=/tmp/merged.dat
SEVERE: Unhandled exception on lcov tool: java.lang.OutOfMemoryError
```
There are a couple of issues:
* The peak working set is quite high. Perhaps ~10x the size of the merged report.
* There is potentially very high GC pressure; even for smaller LCOV files (assuming many are being merged), a large number of "temporary" objects (`LineCoverage`, etc) are created and disposed of.
These largely have the same root cause: the internal coverage representation is inefficient; lots of small objects holding redundant data.
An incremental improvement would be to reduce the size of these objects:
* Both `LineCoverage` and `BranchCoverage` hold a "line number" field, but this is largely redundant since it's already known to `SourceFileCoverage` which has to map line number to coverage data already).
* `LineCoverage` could probably be replaced with `Integer`; at least then the Integer cache might provide some savings.
A larger improvement would be to dramatically change how `SourceFileCoverage` is implemented; e.g. replacing the map of line_number->execution count with an array.
Contributor guide
Assessment
This issue has not been assessed yet.