dotnet / dotnet/xharness

Validate Android failed-test detection across custom runner paths

Open
#1,669 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
189
Forks
67
Avg merge
1d 12h
Merged PRs (30d)
29

Description

## Problem

#1665 makes `xharness android test` and `xharness android run` inspect pulled test-result XML and return `TESTS_FAILED` when the report contains failed tests. This is necessary because not every Android runner reliably propagates the failed-test count through its process or instrumentation exit code.

Some consumers also contain their own failure-propagation logic, while NativeAOT and other custom runner paths may behave differently. The division of responsibility is unclear and can leave untested false-green paths.

Relevant change: #1665.

## Proposed work

- Inventory the standard and custom Android runner paths used with XHarness, including NativeAOT-style runners.
- Verify that each path fails when the pulled xUnit or NUnit report contains failed tests but instrumentation exits successfully.
- Add integration coverage for any currently untested runner path.
- Document that XHarness result analysis is the common final safeguard and identify any consumer-side logic that must remain.
- Remove redundant consumer-side checks only when equivalent XHarness coverage is proven.

## Completion criteria

- Every supported Android runner path has a test demonstrating that failed tests cannot produce a successful XHarness result.
- The ownership of result parsing and exit-code generation is documented.

_This issue was created with AI assistance._

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing change #1665 and the xharness android test and android run entry points, then inventory the standard and custom Android runner paths, including NativeAOT-style runners. Add integration coverage where a pulled xUnit or NUnit result XML reports failures despite a successful instrumentation exit, and document the ownership of result parsing and exit-code generation.

Written by the indexing model from the issue text.

Assessment

Tech stack
android, csharp
Domain
cli, mobile-dev, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.