[infra] Mono FullAOT runtime test runner does not report (sometimes) correct exit code

Open
#118,969 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
csharp
Domain
ci-cd, testing-qa

Research direction

Start by examining the linked Azure DevOps job, its Tests tab, Build Analysis, and the Helix logs for the minifullaot runtime test run. Compare the individual failures with the workitem's reported exit code and trace the merged test runner; done means all failing tests are surfaced and the workitem returns a nonzero exit code.

Written by the indexing model from the issue text.

Description

area-Infrastructure-mono

There appear to be failing tests which are not displayed on Azdo build log, probably because the test suit incorrectly reports exit code 0 even though it contains failing tests inside. This could be related to merged test runner (cc: @jkoritzinsky).

Example:

The https://dev.azure.com/dnceng-public/public/_build/results?buildId=1128424&view=logs&j=9c845561-93d9-5f12-0979-955ea2f35497&t=644e769d-37f3-54b4-c202-c63c42bf5d0c (job linux-x64 Release AllSubsets_Mono_MiniFullAot_RuntimeTests minifullaot) displays only 1 failure: Regression_2

However, looking at the Azdo "Tests" tab, more failures are present

Image

Similarly, Build Analysis also detects the additional failure. After exploring the Helix logs for the additional test failures, failing tests are present

Expected: 100
Actual: 134
END EXECUTION - FAILED

but the workitem reports exit code 0.

Dominant language
C#
Stars
18.3k
Forks
5.6k
PR metrics pending

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from dotnet/runtime

All issues in dotnet/runtime

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.