E2E test coverage doesn't capture coverage of (sub)commands that exit uncleanly

SUMMARY

Since Go doesn't have native support for capturing the coverage of Go programs that are built separately and called as external commands (i.e. in E2E tests), we use a work-around that creates a dummy test (TestCoverageE2E) that executes the main function and put it behind a build tag (e2ecoverage).

Here is an example for the oasis-node command:

https://github.com/oasisprotocol/oasis-core/blob/ffa545427af15995184633455177ea5b1ad4e8af/go%2Foasis-node%2Fcoverage_test.go#L1-L13

This approach is inspired by Filippo Valsorda's blog post on Go Coverage with external tests.

As mentioned in the blog post, this approach has a downside, namely that the command must return from its main function cleanly for the coverage profile to be written to disk. Otherwise, it is just empty.

The consequence in our case is that coverage profiles for external (sub)commands that exit with non-zero exit codes have their test coverage profiles (i.e. coverage-e2e-*.txt files) empty.
Thus, the code that was covered by their runs, e.g. up to a os.Exit(1) call, is not counted towards the reported test coverage.

ISSUE TYPE

• Bug Report

COMPONENT NAME

Go E2E tests.

Start with go/oasis-node/coverage_test.go and the Go E2E coverage setup described in the issue. Reproduce coverage for an external command or subcommand that exits non-zero, then trace how its coverage profile is produced. Done means covered code is retained in coverage-e2e-*.txt files even when the command exits uncleanly.