microsoft / microsoft/vscode-cmake-tools

[Feature] CTest: Extract `TestMessage` properties from test failure output using regular expressions

Open
#4,418 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
TypeScript
Stars
1.7k
Forks
546
Avg merge
2d 16h
Merged PRs (30d)
32

Description

### Request Overview

#### Background

CTest can be used to include tests from C++ unit test frameworks into a project's suite of automated tests. CMake Tools already has support for associating tests with source locations using the `DEF_SOURCE_LINE` property. This property is added to tests by at least [GoogleTest](https://github.com/google/googletest) and [CppUTest](https://cpputest.github.io/).

Beyond that, though, CMake Tools doesn't attempt to examine failure output to determine the source location for test failures. It just always reports the first line of the test as the location of any failures. It also doesn't do anything to take advantage of Test Explorer's`actualOutput`/`expectedOutput` diffing capabilities.

#### Proposed Feature

I'm proposing adding the capability to examine test failure output for source file, source line, message, actual output, and expected output. The format of the output differs between test frameworks, and a single project's test suite could conceivably have heterogeneous output formats for different kinds of tests (smoke, integration, unit, etc.) so it wouldn't make sense to hard-code any particular output parsing. Instead, Regular Expressions with capture groups could be used in a way similar to how Tasks use them in Problem Matcher patterns.

### Additional Information

I have a PR that implements this feature, I'll be posting it soon. I've been trying it out with CppUTest on my projects and it's working great, adding detail and specificity to the results.

The configuration is based on the one used for Problem Matcher patterns, centered around regular expressions and match group indexes. It's not identical to Problem Matchers, though, because of the different needs of this scenario; chiefly, the need to capture expected and actual output.

In my current implementation, the setting `cmake.ctest.failurePatterns` is a list of patterns with the fields `regexp`, `file`, `line`, `message`, `actual`, and `expected`. My settings for CppUTest look like this, for example:
```jsonc
"cmake.ctest.failurePatterns": [
// CHECK_EQUAL gives enough information to allow diffing
{
"regexp": "(.*?):(\\d+):.*\\n(\\texpected *<(.*)>\\n\\tbut was *<(.*)>\\n(?:\\t.*\\n)*)",
"expected": 4,
"actual": 5,
},
// If there are indented lines of text, use them as the message
"(.*?):(\\d+):.*\\n((?:\\t.+\\n)+)",
// Otherwise, fall back to Failure in TEST(...)
"(.*?):(\\d+): ?error: *(.*)\\n(?=[^\\s]|\\t[ \\t]*\n)",
]
```

If more than one pattern is listed, all of them are checked against the test output. This is because CTest can be used to run arbitrary commands, so there could potentially be different tools with different output formats on the same project. I'm not sure whether that makes the most sense, or whether it makes more sense to stop trying patterns on a particular test's output once one matches. It's an easy change to make, at any rate.

This implementation is simpler than Task Problem Patterns in some respects. It doesn't have `character`, `endLine`, or `endCharacter` properties, since C/C++ unit test frameworks tend to only use `__FILE__` and `__LINE__` to generate failure details. It also doesn't have anything like the MultiLineProblemPattern / `loop`feature of Task Problem Matchers. I could add these if there was interest, but I wanted to start with something simple and grow it in response to feedback.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing CTest failure-output handling and the Problem Matcher pattern configuration; the issue names no repository files or tests. Compare the requested failurePatterns fields and capture-group behavior with Test Explorer's actualOutput/expectedOutput needs, then verify that configured patterns produce file, line, message, actual, and expected results.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp, typescript
Domain
devtools, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.