dotnet / dotnet/msbuild

[aw] Flaky Test Auto-Fixer reported incomplete result

Open
#14,981 0 comments 0 reactions 0 assignees View on GitHub
agentic-workflows
Dominant language
C#
Stars
5.5k
Forks
1.5k
Avg merge
1d 8h
Merged PRs (30d)
141

Description

### Workflow Failure

**Workflow:** [Flaky Test Auto-Fixer](https://github.com/dotnet/msbuild/blob/main/.github/workflows/flaky-test-fixer.agent.md)
**Branch:** main
**Run:** https://github.com/dotnet/msbuild/actions/runs/34228749583

> [!WARNING]
> **Task Could Not Be Completed**: The agent reported that the task could not be performed due to an infrastructure or tool failure.

**Reasons:**
- Cannot run the required detector script (.github/workflows/scripts/Get-FlakyTests.ps1) because the sandboxed PowerShell environment fails to load core PowerShell modules (Microsoft.PowerShell.Management, Microsoft.PowerShell.Utility) — even basic cmdlets like Write-Host, New-Item, and Get-ChildItem are unavailable, and direct Assembly.LoadFrom() on the module DLLs fails with FileNotFoundException despite the files existing on disk. This is blocked before the DaysBack/MinSources scan step can even build the temp directory, so Step 1 of the workflow cannot execute.

Environment: pwsh 7.6.5 on Linux (Ubuntu, GitHub Actions runner). `pwsh -c '1+1'` works and basic .NET types (System.IO.Directory.CreateDirectory) work fine, but PowerShell's binary/script modules under /opt/microsoft/powershell/7/Modules/ (which exist on disk with correct permissions, verified via ls/file) cannot be imported — Import-Module and even direct Reflection.Assembly.LoadFrom() on their DLLs both throw FileNotFoundException, and no cmdlets they provide (Write-Host, New-Item, Get-ChildItem, Join-Path, Select-Object, Format-List) are available. This looks like a sandbox/apparmor-level restriction on loading additional native/managed binaries beyond the initially launched pwsh process (aa-status confirms apparmor is active but unreadable without privilege). Attempted workarounds: setting TMPDIR/TMP/TEMP env vars (irrelevant — module loading fails regardless), explicit Import-Module with -Force/-ErrorAction Stop, direct assembly loading. All failed identically. Without functioning PowerShell modules, the Get-FlakyTests.ps1 script cannot create its temp directory (New-Item) or execute any of its logic (Invoke-RestMethod, ConvertFrom-Json, etc. would also be affected), so Step 1 (scanning quarantine pipeline definition 344) cannot be performed, and per the workflow's guard rails, no diagnosis or PR should be attempted without real scan data.

This is a structured incompletion signal (`report_incomplete`), not a real task outcome. Any other safe outputs emitted alongside this signal (e.g., comments) describe the failure state, not a completed review or action.

### Action Required

**Assign this issue to an agent** to debug and fix the issue.

Debug with any coding agent

Use this prompt with any coding agent (GitHub Copilot, Claude, Gemini, etc.):

````
Debug the agentic workflow failure using https://raw.githubusercontent.com/github/gh-aw/main/debug.md

The failed workflow run is at https://github.com/dotnet/msbuild/actions/runs/34228749583
````

Manually invoke the agent

Debug this workflow failure using your favorite Agent CLI and the `agentic-workflows` prompt.

- Start your agent
- Load the `agentic-workflows` skill from `.github/skills/agentic-workflows/SKILL.md` or
- Type `debug the agentic workflow flaky-test-fixer.agent failure in https://github.com/dotnet/msbuild/actions/runs/34228749583`

> [!TIP]
>
> Stop reporting this workflow as a failure
>
> To stop a workflow from creating failure issues, set `report-failure-as-issue: false` in its frontmatter:
> ```yaml
> safe-outputs:
> report-failure-as-issue: false
> ```
>
>

> Generated from [Flaky Test Auto-Fixer](https://github.com/dotnet/msbuild/actions/runs/34228749583) · copilot · 51.7 AIC · [◷](https://github.com/search?q=repo%3Adotnet%2Fmsbuild+is%3Aissue+%22gh-aw-workflow-id%3A+flaky-test-fixer.agent%22&type=issues)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with .github/workflows/flaky-test-fixer.agent.md and .github/workflows/scripts/Get-FlakyTests.ps1, then inspect the failed run linked in the issue. Reproduce the pwsh 7.6.5 module-loading failure on the Ubuntu runner and verify the detector can execute its scan. Done means the workflow runs Step 1 and produces real scan data without reporting infrastructure incompletion.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, powershell
Domain
ci-cd, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.