dotnet / dotnet/roslyn

Flaky test: AutoLoadProjectsTests.RestoresSolutionInsteadOfIndividualProjectsWhenSolutionLoaded -- "Restoring App.csproj" instead of "Restoring App.sln"

Open
#85,295 1 comment 0 reactions 0 assignees View on GitHub
Area-IDE
Dominant language
C#
Stars
20.7k
Forks
4.3k
PR merge metrics
PR metrics pending

Description

## Summary

`Microsoft.CodeAnalysis.LanguageServer.ProcessHost.UnitTests.Workspaces.AutoLoadProjectsTests.RestoresSolutionInsteadOfIndividualProjectsWhenSolutionLoaded` failed intermittently on Linux Debug CI. In plain terms: the test loads a solution containing two unrestored projects and expects a single restore of the whole solution (`Restoring App.sln`). Instead, only one project (`App.csproj`) was restored on its own (`Restoring App.csproj`), and that restore actually succeeded -- this isn't a broken restore, it's the "restore only what's needed" logic kicking in on just one project before the second project had been queued.

## Failure details

- Build: [1587906](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1587906) (roslyn-CI, branch `refs/pull/85028/merge`, PR #85028)
- Failed step (attempt 1): [Test / Test_Linux_Debug](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=1587906&view=logs&t=b2948985-b3d8-5780-7b2b-b582fd83c893&attempt=1&j=644ea285-4692-54f6-2965-667c9562a0e2)
- Test result: [run 43825534, result 248620](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/test/runs/43825534/results/248620?api-version=7.1)
- The same build's stage attempt 2 (job `Test_Linux_Debug`) passed on retry, including this test's assembly.

```text
Assert.Equal() Failure: Strings differ
v (pos 14)
Expected: "Restoring App.sln"
Actual: "Restoring App.csproj"
^ (pos 14)
at Microsoft.CodeAnalysis.LanguageServer.ProcessHost.UnitTests.Workspaces.AutoLoadProjectsTests.RestoresSolutionInsteadOfIndividualProjectsWhenSolutionLoaded() in /_/src/LanguageServer/Microsoft.CodeAnalysis.LanguageServer.ProcessHost.UnitTests/Workspaces/AutoLoadProjectsTests.cs:line 161
```

The retained xUnit diagnostic log for the failed attempt (`xUnitFailure-Microsoft.CodeAnalysis.LanguageServer.ProcessHost.UnitTests_36.log`) shows:
- `App.sln` was found and auto-loaded.
- Only `App.csproj` was loaded/reloaded ("Completed (re)load of 1 project(s)") before the restore kicked off.
- `dotnet restore` ran against `App.csproj` alone and **completed successfully**.
- `Nested.csproj` never appears as loaded in the log before the test tore down the server.

## Root cause candidates (unconfirmed -- need to pick between these)

1. **Product timing issue**: `App.csproj` and `Nested.csproj` are queued into the project loader's `AsyncBatchingWorkQueue` (100ms batch window, see `LanguageServerProjectLoader.cs`) sequentially as the solution is opened. If the first batch fires before the second project's load is enqueued, `GetPathsToRestoreAsync` (`LanguageServerProjectSystem.cs`) sees only one project needing restore and -- per the singleton-restore optimization added in commit `6ab5e880562ec35a626d9c7690566fe10244f232` ("Don't restore the entire solution if we still have only one project") -- restores that project individually instead of waiting to coalesce with the rest of the solution. If both projects loading into the *same* batch is supposed to be a reliable guarantee once a solution is open, this is a genuine race in project enqueueing/batching that should be fixed in the product.

2. **Test assumption is too strict**: the singleton-restore optimization is intentional (added deliberately, not accidentally), and it may be entirely acceptable/expected for the two projects to sometimes load in separate batches under CI-level scheduling variance. In that case the test itself is asserting something the product never actually guarantees, and the fix belongs in the test (e.g., wait for solution load to fully settle before asserting, or tolerate a per-project restore for projects that load before the rest of the solution).

## Suggested next step

Instrument (or otherwise determine) exactly when `App.csproj` and `Nested.csproj` are each enqueued relative to the loader's 100ms batching window during `OpenSolutionAsync`. That will show whether this is a genuine ordering race the product should guard against, or an expected multi-batch scenario that the test needs to tolerate.

---
This was filed based on triage of a single observed CI failure; recurrence/frequency has not yet been separately measured.

Contributor guide

Open the contributing guide

Research direction

Start with AutoLoadProjectsTests.cs at line 161 and trace OpenSolutionAsync through LanguageServerProjectLoader.cs and LanguageServerProjectSystem.cs. Instrument when App.csproj and Nested.csproj enter the AsyncBatchingWorkQueue relative to its 100ms window, then determine whether product batching or the test assumption is at fault. Done means the race is fixed or the test reliably reflects the intended restore behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.