Ability to verify that all library tests can be run locally without failures
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- csharp
- Domain
- build-system, ci-cd, testing
Research direction
Start with the documented command build -s clr+libs+libs.tests -rc release -test and the listed failing or hanging projects, including HttpListener.Tests, System.Net.Security.Tests, and System.Security.Cryptography.Tests. Define the local and CI test entry points and verify that the complete library test set runs without failures or hangs, from a non-elevated prompt and without conflicting parallel tests.
Written by the indexing model from the issue text.
Description
The ability to run all library tests locally without failures should be supported, even though it is time-consuming and somewhat rare to want to do this.
I would like to do this before and after making lower-level changes such as for reflection or other runtime-overlapping changes. There are other reasons to want this including verifying that the environment is correct.
Without this, I depend on CI to catch library test failures (and runtime test failures - more on this later).
Expect: add test and CI infrastructure to run all inner loop library tests as a single unit in order to emulate the local results. Perhaps this would run in the outerloop pipeline since it is time consuming. Having CI support is necessary to prevent future breaks. This should also be done in a non-elevated command prompt. In some cases, having multiple test projects run at the same time can result in conflicts -- especially networking tests -- so there may still be flakiness if we don't have a way to prevent that from occurring.
Currently for me, running this on Windows
build -s clr+libs+libs.tests -rc release -test
results in several failures and a hang:
System.DirectoryServices.Tests Total: 255, Errors: 0, Failed: 1, Skipped: 29, Time: 2.878s
System.Net.Quic.Functional.Tests Total: 441, Errors: 0, Failed: 1, Skipped: 1, Time: 189.488s
System.Security.Cryptography.Tests Total: 10125, Errors: 0, Failed: 19, Skipped: 114, Time: 71.050s
System.Net.Security.Tests Total: 5177, Errors: 0, Failed: 27, Skipped: 21, Time: 50.719s
(and then a hang on HttpListener.Tests)
System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerResponseHeadersTests.StatusDescription_GetWithCustomStatusCode_ReturnsExpected', Elapsed: 00:02:07
(running again I get another hang)
System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerRequestTests.ProtocolVersion_GetProperty_ReturnsExpected', Elapsed: 00:08:03
Longer-term
The ideal solution would be to easily specify a "smokescreen" set of tests. For starting, this would be projects that catch lower-level runtime issues, which me for would include specific libraries including System.Runtime, System.Text.Json, System.Runtime.Serialization, System.ComponentModel.TypeConverter and Microsoft.Extensions.DependencyInjection. Today, this can be emulated using && to combine multiple commands, but perhaps this could be done in a single command that would allow parallelization.
Even better, the ability to specify a set of runtime tests along with this would be convenient, although today the runtime test infrastructure is different from the library test infrastructure -- e.g. how to build, how to run, and concepts such as test priority. So doing this in an integrated way may not be feasible unless we have a way to share or wrap these two test infrastructures with similar commands.
- Dominant language
- C#
- Stars
- 18.3k
- Forks
- 5.6k
- PR merge metrics
- PR metrics pending
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from dotnet/runtime
-
agentic-workflows untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
area-System.Reflection blocking-clean-ci-optional Known Build Error os-mac-os-x untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
area-CodeGen-coreclr untriaged
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
agentic-workflows untriaged
Difficulty 1/5 Under an hour Newbie friendliness 78/100
-
area-VM-meta-mono untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
-
:watch: Not Triaged 11.0 fundamentals/subsvc
Difficulty 2/5 1-3 hours Newbie friendliness 92/100
dotnet/AspNetCore.Docs#37699 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
SubtitleEdit/subtitleedit#15108 · 1 comment ·
-
area/docs-content Bug pulumi/docs
Difficulty 1/5 1-3 hours Newbie friendliness 94/100
-
Create parent directories only after the containment check in InstallHelper.TryExtractToDirectory Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
PowerShell/PSResourceGet#2056 ·