[Bug]: Build - Intermittent unit test failure from RunnerUtilities.RunProcessAndGetOutput() with stdout
- Dominant language
- C#
- Stars
- 5.5k
- Forks
- 1.5k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 133
Description
### Issue Description
This is an issue in the unit tests for the project.
There appears to be an issue with the unit test support method `Microsoft.Build.UnitTests.Shared.RunnerUtilities.RunProcessAndGetOutput()`. Intermittently on Windows, a unit test may fail to find the expected output. It appears that `stdout` is not completely captured before the process exits.
### Steps to Reproduce
Currently there are no unit tests for the `/preprocess` and `/targets` switches that test the functionality[^1]. As part of the work for #7697 I am adding unit tests for the `/preprocess` and `/targets` switches for project files and for solution files.
I added the following unit test to `src\MSBuild.UnitTests\XMake_Tests.cs`:
```csharp
///
/// Given an empty project, the Preprocess switch should report an empty project.
///
[Fact]
public void PreprocessSwitchWithEmptyProject()
{
string[] arguments = { @"/pp", @"/nologo" };
string project = @"" + Environment.NewLine + @"" + Environment.NewLine;
string logContents = ExecuteMSBuildExeExpectSuccess(project, arguments: arguments);
logContents.ShouldContain(@"");
}
```
When running all unit tests from the command line (`build.cmd -test`) or in the Visual Studio Test Explorer, this unit test will sometimes fail on the `logContents.ShouldContain(@"")` verification because the output is missing from the string returned by `ExecuteMSBuildExeExpectSuccess`.
The call chain is
`ExecuteMSBuildExeExpectSuccess` -->
`ExecuteMSBuildExe` -->
`RunnerUtilities.ExecMSBuild(string, out bool, ITestOutputHelper)` -->
`RunnerUtilities.ExecMSBuild(string, out bool, bool = false, ITestOutputHelper = null)` -->
`RunnerUtilities.RunProcessAndGetOutput`
[^1]: There are unit tests that test that the switches are recognized as part of handling the command line.
### Expected Behavior
The unit test is expected to always succeed.
### Actual Behavior
Intermittently the unit test will falsely fail.
(Unfortunately, the example unit test will probably not show the issue on the first attempt. :slightly_frowning_face:)
### Analysis
The issue happens on Windows 11 with both net7.0 and net472. On macOS 12 (Monterey) the issue doesn't present. I have not tested on Liunux.
The `/preprocess` and `/targets` switches differ from building a project because they write directly to `Console.Out` and don't use the configured logger(s).
In `XMake.cs`, as an experiment, I added explicit `Flush()` calls after the completion of the work for Preprocess and for Targets. This seemed to lessen the occurrence of the issue but it did not resolve the issue.
### Versions & Configurations
_No response_
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with RunnerUtilities.RunProcessAndGetOutput() and follow the call chain from ExecuteMSBuildExeExpectSuccess in src\MSBuild.UnitTests\XMake_Tests.cs; run build.cmd -test on Windows to observe the intermittent failure. Review the related Console.Out handling in XMake.cs. Done means the preprocess and targets tests consistently receive complete stdout and pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- build-system, operating-systems, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100