Terminated Agent Process leads to lost Test Results
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 234
- Forks
- 163
- Avg merge
- 13h 32m
- Merged PRs (30d)
- 25
Description
In our observation, if the test agent process crashes (e.g. due to an unmanaged exception in a test case, or simulated by an Environment.Exit()), the test results of the whole test run are lost. The TestResult.xml file contains only the exception that the connection to the agent process was interrupted.
Sample Testcase:
[Test]
public void MyTestmethod()
{
Environment.Exit(1);
}
Outcome:
System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
Test Run Summary
Overall result: Failed
Test Count: 0, Passed: 0, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
Is this behavior intended?
We are working in a quite big project, where a test run may consist of several test assemblies with several 10K test cases. Even if the test case which leads to the crash is the last test case in the last test assembly, all previous test results are lost.
Therefore, we are currently searching for a possibility to handle terminations of the nunit agent process without loosing the information of the already executed test cases.
Is there any such possibility?
Can NUnit be extended somehow such that it is able to continuously log the test progress?
The use cases which justify this requirement would be in our case the following:
UseCase 1:
As a developer, I want to be able to analyze the root cause why a test agent has crashed. To do this, I need to know
a) at what point within the test case the crash happened.
b) what are the outcomes of previously run test cases.
UseCase 2:
As an Integrator, I want to be able to kill a running test agent in certain situations (process hangs, …) without loosing the results of the already executed tests in the current test run.
Additional Information:
• We run our tests with process model “Separate” (--process=Separate)
• NUnit Framework Version: 3.10.1
• NUnit Console Version: 3.8.0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the failure with NUnit Framework 3.10.1, Console 3.8.0, and --process=Separate using the Environment.Exit(1) sample. Inspect how the agent termination is represented in TestResult.xml and determine how previously completed tests could remain available after the connection is interrupted; done means results are preserved for a crashed or deliberately killed agent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- cli, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100