dotnet / dotnet/msbuild

[Bug]: Build - Intermittent unit test failure from RunnerUtilities.RunProcessAndGetOutput() with stdout

Open
#9,075 0 comments 0 reactions 0 assignees View on GitHub
Area: Tests bug Priority:2 triaged
Dominant language
C#
Stars
5.5k
Forks
1.5k
Avg merge
1d 13h
Merged PRs (30d)
133

Description

### Issue Description

This is an issue in the unit tests for the project.

There appears to be an issue with the unit test support method `Microsoft.Build.UnitTests.Shared.RunnerUtilities.RunProcessAndGetOutput()`. Intermittently on Windows, a unit test may fail to find the expected output. It appears that `stdout` is not completely captured before the process exits.

### Steps to Reproduce

Currently there are no unit tests for the `/preprocess` and `/targets` switches that test the functionality[^1]. As part of the work for #7697 I am adding unit tests for the `/preprocess` and `/targets` switches for project files and for solution files.

I added the following unit test to `src\MSBuild.UnitTests\XMake_Tests.cs`:

```csharp
///
/// Given an empty project, the Preprocess switch should report an empty project.
///
[Fact]
public void PreprocessSwitchWithEmptyProject()
{
string[] arguments = { @"/pp", @"/nologo" };
string project = @"" + Environment.NewLine + @"" + Environment.NewLine;
string logContents = ExecuteMSBuildExeExpectSuccess(project, arguments: arguments);
logContents.ShouldContain(@"");
}
```

When running all unit tests from the command line (`build.cmd -test`) or in the Visual Studio Test Explorer, this unit test will sometimes fail on the `logContents.ShouldContain(@"")` verification because the output is missing from the string returned by `ExecuteMSBuildExeExpectSuccess`.

The call chain is
`ExecuteMSBuildExeExpectSuccess` -->
`ExecuteMSBuildExe` -->
`RunnerUtilities.ExecMSBuild(string, out bool, ITestOutputHelper)` -->
`RunnerUtilities.ExecMSBuild(string, out bool, bool = false, ITestOutputHelper = null)` -->
`RunnerUtilities.RunProcessAndGetOutput`

[^1]: There are unit tests that test that the switches are recognized as part of handling the command line.

### Expected Behavior

The unit test is expected to always succeed.

### Actual Behavior

Intermittently the unit test will falsely fail.

(Unfortunately, the example unit test will probably not show the issue on the first attempt. :slightly_frowning_face:)

### Analysis

The issue happens on Windows 11 with both net7.0 and net472. On macOS 12 (Monterey) the issue doesn't present. I have not tested on Liunux.

The `/preprocess` and `/targets` switches differ from building a project because they write directly to `Console.Out` and don't use the configured logger(s).

In `XMake.cs`, as an experiment, I added explicit `Flush()` calls after the completion of the work for Preprocess and for Targets. This seemed to lessen the occurrence of the issue but it did not resolve the issue.

### Versions & Configurations

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with RunnerUtilities.RunProcessAndGetOutput() and follow the call chain from ExecuteMSBuildExeExpectSuccess in src\MSBuild.UnitTests\XMake_Tests.cs; run build.cmd -test on Windows to observe the intermittent failure. Review the related Console.Out handling in XMake.cs. Done means the preprocess and targets tests consistently receive complete stdout and pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
build-system, operating-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.