C++ coverage performance
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
Envoy continues to experiment with switching away from its own custom coverage solution to native `bazel coverage`, but we have two stumbling blocks
1. https://github.com/bazelbuild/bazel/issues/7279 <-- we need to be able to use remote caching (and ideally full RBE in the near future) to realize build time gains.
2. Test execution is too slow. It's slow enough that the parallelism we gain from executing all tests in parallel vs. our single monolithic binary today is no win.
Investigating (2) in https://github.com/envoyproxy/envoy/pull/6703 has some interesting possible improvements. We see that performance when we build all targets `//test/...` as about 2h 8 m for build+test on CI, which is a bit slower than our existing single test binary solution which clocks in at around < 2h . This should probably be much faster, as test execution can now be split across cores.
I think that what's going on is that `collect_cc_coverage.sh` has major performance issues with `gcov` (let alone the `lcov` trace merger which we aren't hitting yet but is pretty bad based on historical performance work https://github.com/bazelbuild/bazel/issues/1118#issuecomment-292830621).
I instrumented and measured test vs. collection time in https://github.com/envoyproxy/envoy/pull/6703 for `CC=gcc CXX=g++ COVERAGE_TARGET=//test/common/upstream:cluster_manager_impl_test VALIDATE_COVERAGE=false ./test/run_envoy_bazel_coverage.sh`. This yielded:
* Test time: 2.87s
* `collect_cc_coverage.sh` time: 43.56s
What's going on I think is the performance of this loop is to blame: https://github.com/bazelbuild/bazel/blob/9a32b861799278217c37eb32f864a951ca99617e/tools/test/collect_cc_coverage.sh#L87.
Iterating over all the files in a large tree for each test in shell is pretty slow. Maybe we could have a high performance C++ gcov trace merger?
Ideally we switch to `lcov`/`clang` going forward, but this will require the low performance `geninfo` to be replaced to make it practical for Envoy.
CC @mattklein123 @lizan @iirina
Contributor guide
Research direction
Start with tools/test/collect_cc_coverage.sh at the linked loop and reproduce the reported command with CC=gcc, CXX=g++, COVERAGE_TARGET=//test/common/upstream:cluster_manager_impl_test, and VALIDATE_COVERAGE=false. Measure test time versus gcov collection time, then compare the result with the existing single-binary coverage approach. Done means the coverage workflow no longer spends most of its time collecting traces and improves on the reported performance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, shell
- Domain
- build-system, performance, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100