dotnet / dotnet/sdk

CA CLI invocation tests flake on Mac arm64 Helix due to `-tlp:DISABLENODEDISPLAY` leaking through `AdjustMSBuildForLLMs`

Open
#56,258 3 comments 1 reaction 3 assignees Claimed by @JeremyKuhne View on GitHub
Area-CLI Bug cookie Known Build Error Test Debt
Dominant language
C#
Stars
3.2k
Forks
1.3k
PR merge metrics
PR metrics pending

Description

> [!NOTE]
> This issue was generated with the help of AI.

## Build Information
Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1592515
Leg Name: `dotnet-sdk-public-ci (Build Mac arm64)` — Helix work item `dotnet.Tests.dll.3` in job `Mac arm64 - osx.15.arm64.open`
Observed on PR: #54127 (unrelated analyzer allocation change — the same three tests fail here because of the environment, not the diff)

## Failing tests

Three tests in `dotnet.Tests` on Mac arm64 assert exact MSBuild forwarding arg lists and now trip over an unexpected `-tlp:DISABLENODEDISPLAY` token:

- `Microsoft.DotNet.Cli.MSBuild.Tests.GivenDotnetPublishInvocation.MsbuildInvocationIsCorrect(["--configuration",""], ["--property:Configuration="])`
- `Microsoft.DotNet.Cli.MSBuild.Tests.GivenDotnetPublishInvocation.MsbuildInvocationIsCorrectForNoBuild`
- `Microsoft.DotNet.Cli.MSBuild.Tests.GivenDotnetStoreInvocation.ItAddsOutputPathToMsBuildInvocation("-o")` (test file: `test/dotnet.Tests/CommandTests/MSBuild/GivenDotnetStoreInvocation.cs`)

## Root cause (proximate)

`MSBuildForwardingApp` unconditionally routes its arg list through `CommonRunHelpers.AdjustMSBuildForLLMs` (`src/Cli/dotnet/Commands/MSBuild/MSBuildForwardingApp.cs:90`), which appends `Constants.TerminalLogger_DisableNodeDisplay` (`-tlp:DISABLENODEDISPLAY`) whenever `LLMEnvironmentDetectorForTelemetry.IsLLMEnvironment()` returns true (`src/Cli/dotnet/Commands/Run/CommonRunHelpers.cs:27`). The detector inspects a large list of environment variables (`CLAUDECODE`, `COPILOT_*`, `CODEX_*`, `CURSOR_*`, `AGENT_CLI`, …) and returns true if any are set (`src/Cli/dotnet/Telemetry/LLMEnvironmentDetectorForTelemetry.cs`). Something in the Mac arm64 Helix work-item environment is setting one of these variables, so the invocation-shape assertions get an extra token and fail.

The three tests do not scrub these environment variables before constructing the forwarding app, and their `expectedAdditionalArgs` arrays do not account for the LLM-mode adjustment, so they are not stable against ambient environment.

## Failure output (excerpt)

```
failed ItAddsOutputPathToMsBuildInvocation ("-o") (10ms)
FluentAssertions.Execution.AssertionFailedException: Expected subject to be a collection with 8 item(s), but
{"-maxcpucount", "--verbosity:m", "-tlp:default=auto",
"--property:ComposeDir=/private/tmp/helix/working/AD460949/w/B39809C5/e/some/path/",
"--property:_CommandLineDefinedOutputPath=true", "--target:ComposeStore", "--nologo", "",
"-tlp:DISABLENODEDISPLAY"}

failed MsbuildInvocationIsCorrectForNoBuild (14ms)
FluentAssertions.Execution.AssertionFailedException: Expected subject to be a collection with 8 item(s), but
{"-maxcpucount", "--verbosity:m", "-tlp:default=auto", "--property:NoBuild=true",
"--property:NuGetInteractive=false", "--property:_IsPublishing=true", "--target:Publish", "--nologo",
"-tlp:DISABLENODEDISPLAY"}

failed MsbuildInvocationIsCorrect (["--configuration",""], ["--property:Configuration="])
... items {"-tlp:DISABLENODEDISPLAY"} are not part of the superset.
```

Console log: https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-54127-merge-3b41a699ba7241eebf/dotnet.Tests.dll.3/1/console.39406bda.log?helixlogtype=result

## Suggested fix

Make the three (and any peer) invocation-shape tests deterministic by either
(a) unsetting the LLM-detector environment variables in a test fixture before constructing `MSBuildForwardingApp`, or
(b) accepting `-tlp:DISABLENODEDISPLAY` as an optional trailing arg in the expected arg set, or
(c) injecting an `ILLMEnvironmentDetector` seam so tests can force detection off.

The `AdjustMSBuildForLLMs` helper was introduced in #51054.

## Error Message
```json
{
"ErrorMessage": "-tlp:DISABLENODEDISPLAY",
"BuildRetry": false,
"ExcludeConsoleLog": false
}
```

### Known issue validation
**Build: :mag_right:** https://dev.azure.com/dnceng-public/public/_build/results?buildId=1592515
**Error message validated:** `[-tlp:DISABLENODEDISPLAY`]
**Result validation:** :white_check_mark: Known issue matched with the provided build.
**Validation performed at:** 9/11/2026 1:04:49 PM UTC

### Report

|Build|Repository|Test|Pull Request|
|---|---|---|---|
|[1592515](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1592515)|dotnet/sdk|[Microsoft.DotNet.Cli.MSBuild.Tests.GivenDotnetPublishInvocation.MsbuildInvocationIsCorrectForNoBuild](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1592515&view=ms.vss-test-web.build-test-results-tab&runId=43960748&resultId=110195)|dotnet/sdk#54127|
|[1591740](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1591740)|dotnet/sdk|[Microsoft.DotNet.Cli.MSBuild.Tests.GivenDotnetPublishInvocation.MsbuildInvocationIsCorrectForNoBuild](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1591740&view=ms.vss-test-web.build-test-results-tab&runId=43938430&resultId=104252)|dotnet/sdk#56242|
#### Summary
|24-Hour Hit Count|7-Day Hit Count|1-Month Count|
|---|---|---|
|2|2|2|

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.