Ability to verify that all library tests can be run locally without failures

Open
#111,721 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
csharp

Research direction

Start with the documented command build -s clr+libs+libs.tests -rc release -test and the listed failing or hanging projects, including HttpListener.Tests, System.Net.Security.Tests, and System.Security.Cryptography.Tests. Define the local and CI test entry points and verify that the complete library test set runs without failures or hangs, from a non-elevated prompt and without conflicting parallel tests.

Written by the indexing model from the issue text.

Description

area-Infrastructure-libraries feature-request

The ability to run all library tests locally without failures should be supported, even though it is time-consuming and somewhat rare to want to do this.

I would like to do this before and after making lower-level changes such as for reflection or other runtime-overlapping changes. There are other reasons to want this including verifying that the environment is correct.

Without this, I depend on CI to catch library test failures (and runtime test failures - more on this later).

Expect: add test and CI infrastructure to run all inner loop library tests as a single unit in order to emulate the local results. Perhaps this would run in the outerloop pipeline since it is time consuming. Having CI support is necessary to prevent future breaks. This should also be done in a non-elevated command prompt. In some cases, having multiple test projects run at the same time can result in conflicts -- especially networking tests -- so there may still be flakiness if we don't have a way to prevent that from occurring.

Currently for me, running this on Windows

build -s clr+libs+libs.tests -rc release -test 

results in several failures and a hang:

System.DirectoryServices.Tests  Total: 255, Errors: 0, Failed: 1, Skipped: 29, Time: 2.878s
System.Net.Quic.Functional.Tests  Total: 441, Errors: 0, Failed: 1, Skipped: 1, Time: 189.488s
System.Security.Cryptography.Tests  Total: 10125, Errors: 0, Failed: 19, Skipped: 114, Time: 71.050s
System.Net.Security.Tests  Total: 5177, Errors: 0, Failed: 27, Skipped: 21, Time: 50.719s

(and then a hang on HttpListener.Tests)
System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerResponseHeadersTests.StatusDescription_GetWithCustomStatusCode_ReturnsExpected', Elapsed: 00:02:07
(running again I get another hang)
System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerRequestTests.ProtocolVersion_GetProperty_ReturnsExpected', Elapsed: 00:08:03
Longer-term

The ideal solution would be to easily specify a "smokescreen" set of tests. For starting, this would be projects that catch lower-level runtime issues, which me for would include specific libraries including System.Runtime, System.Text.Json, System.Runtime.Serialization, System.ComponentModel.TypeConverter and Microsoft.Extensions.DependencyInjection. Today, this can be emulated using && to combine multiple commands, but perhaps this could be done in a single command that would allow parallelization.

Even better, the ability to specify a set of runtime tests along with this would be convenient, although today the runtime test infrastructure is different from the library test infrastructure -- e.g. how to build, how to run, and concepts such as test priority. So doing this in an integrated way may not be feasible unless we have a way to share or wrap these two test infrastructures with similar commands.

Dominant language
C#
Stars
18.3k
Forks
5.6k
PR merge metrics
PR metrics pending

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from dotnet/runtime

All issues in dotnet/runtime

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.