nunit / nunit/nunit-console

Terminated Agent Process leads to lost Test Results

Open
#413 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Enhancement hardfix Needs Design Needs Discussion V4Plus
Dominant language
C#
Stars
234
Forks
163
Avg merge
13h 32m
Merged PRs (30d)
25

Description

In our observation, if the test agent process crashes (e.g. due to an unmanaged exception in a test case, or simulated by an Environment.Exit()), the test results of the whole test run are lost. The TestResult.xml file contains only the exception that the connection to the agent process was interrupted.

Sample Testcase:

[Test]
public void MyTestmethod()
{
    Environment.Exit(1);
}

Outcome:

System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

Test Run Summary
  Overall result: Failed
  Test Count: 0, Passed: 0, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0

Is this behavior intended?

We are working in a quite big project, where a test run may consist of several test assemblies with several 10K test cases. Even if the test case which leads to the crash is the last test case in the last test assembly, all previous test results are lost.
Therefore, we are currently searching for a possibility to handle terminations of the nunit agent process without loosing the information of the already executed test cases.
Is there any such possibility?
Can NUnit be extended somehow such that it is able to continuously log the test progress?

The use cases which justify this requirement would be in our case the following:

UseCase 1:
As a developer, I want to be able to analyze the root cause why a test agent has crashed. To do this, I need to know
a) at what point within the test case the crash happened.
b) what are the outcomes of previously run test cases.

UseCase 2:
As an Integrator, I want to be able to kill a running test agent in certain situations (process hangs, …) without loosing the results of the already executed tests in the current test run.

Additional Information:
• We run our tests with process model “Separate” (--process=Separate)
• NUnit Framework Version: 3.10.1
• NUnit Console Version: 3.8.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with NUnit Framework 3.10.1, Console 3.8.0, and --process=Separate using the Environment.Exit(1) sample. Inspect how the agent termination is represented in TestResult.xml and determine how previously completed tests could remain available after the connection is interrupted; done means results are preserved for a crashed or deliberately killed agent.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
cli, testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.