Validate Android failed-test detection across custom runner paths
- Dominant language
- C#
- Stars
- 189
- Forks
- 67
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 29
Description
## Problem
#1665 makes `xharness android test` and `xharness android run` inspect pulled test-result XML and return `TESTS_FAILED` when the report contains failed tests. This is necessary because not every Android runner reliably propagates the failed-test count through its process or instrumentation exit code.
Some consumers also contain their own failure-propagation logic, while NativeAOT and other custom runner paths may behave differently. The division of responsibility is unclear and can leave untested false-green paths.
Relevant change: #1665.
## Proposed work
- Inventory the standard and custom Android runner paths used with XHarness, including NativeAOT-style runners.
- Verify that each path fails when the pulled xUnit or NUnit report contains failed tests but instrumentation exits successfully.
- Add integration coverage for any currently untested runner path.
- Document that XHarness result analysis is the common final safeguard and identify any consumer-side logic that must remain.
- Remove redundant consumer-side checks only when equivalent XHarness coverage is proven.
## Completion criteria
- Every supported Android runner path has a test demonstrating that failed tests cannot produce a successful XHarness result.
- The ownership of result parsing and exit-code generation is documented.
_This issue was created with AI assistance._
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing change #1665 and the xharness android test and android run entry points, then inventory the standard and custom Android runner paths, including NativeAOT-style runners. Add integration coverage where a pulled xUnit or NUnit result XML reports failures despite a successful instrumentation exit, and document the ownership of result parsing and exit-code generation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- android, csharp
- Domain
- cli, mobile-dev, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100