dotnet / dotnet/android

Migrate on-device APK tests from NUnit to MSTest

Open
#12,721 0 comments 1 reaction 0 assignees View on GitHub
needs-triage
Dominant language
C#
Stars
2.1k
Forks
579
Avg merge
1d 20h
Merged PRs (30d)
257

Description

### Android framework version

net11.0-android (Preview)

### Affected platform version

.NET 11 / MSTest 4.4.0

### Description

MSTest 4.4.0 supports NativeAOT: https://www.nuget.org/packages/MSTest/4.4.0

Migrate all on-device APK tests from NUnit to MSTest. This scope includes:

- `tests/Mono.Android-Tests/Mono.Android-Tests/Mono.Android.NET-Tests.csproj`, including tests loaded from `Java.Interop-Tests.NET.csproj`
- `tests/CodeGen-Binding/Xamarin.Android.JcwGen-Tests/Xamarin.Android.JcwGen-Tests.csproj`
- The shared Android instrumentation runner in `tests/TestRunner.Core/`
- Related CI configuration, test documentation, and package/version configuration

The current shared runner directly uses NUnit APIs for discovery, filtering, execution, result counting, and TRX generation. The migration must preserve:

- Android instrumentation startup and the `Microsoft.Android.Run` result protocol
- Multiple test assembly support
- Include/exclude category behavior from instrumentation arguments and `RuntimeHostConfigurationOption`
- Runtime-specific exclusions for Mono, CoreCLR, trimmable type maps, LLVM, and NativeAOT
- Fully-qualified test-name exclusions for external test assemblies
- Execution on the instrumentation/JNI-attached thread where required
- `dotnet test`, MTP, and TRX reporting behavior

The NativeAOT configuration currently contains NUnit-specific reflection/rooting workarounds (`IlcGenerateCompleteTypeMetadata` and `NativeAOT.rd.xml`, including roots for NUnit async adapters). Remove any workarounds that become unnecessary after the migration.

This issue does **not** include host-side NUnit test projects such as `MSBuildDeviceIntegration` or the Android SDK/build-task unit tests.

#### Acceptance criteria

- The two on-device APK test applications and `TestRunner.Core` no longer reference or use NUnit.
- Existing on-device tests are migrated to MSTest without losing coverage; any intentional semantic differences are documented.
- Existing include/exclude filters and runtime-specific skip behavior continue to work.
- The following CI lanes continue to build, install, run, and publish results:
- `Mono.Android.NET_Tests-Debug`
- `Mono.Android.NET_Tests-Release`
- `Mono.Android.NET_Tests-NoAab`
- `Mono.Android.NET_Tests-Mono`
- `Mono.Android.NET_Tests-CoreCLRTrimmable`
- `Mono.Android.NET_Tests-NativeAOT`
- `Xamarin.Android.JcwGen_Tests`
- Local documented `dotnet-local` build/test commands continue to work and produce TRX results.
- NUnit-specific NativeAOT metadata/rooting workarounds are removed when they are no longer required.

### Steps to Reproduce

1. Inspect `tests/TestRunner.Core/TestInstrumentation.cs`; it uses `NUnitTestAssemblyRunner` and NUnit filters/results directly.
2. Inspect the two APK test projects; both reference NUnit and use NUnit test attributes/assertions.
3. Build the NativeAOT lane with `-p:TestsFlavor=NativeAOT -p:PublishAot=true`; the project currently requires NUnit-specific runtime directives.

/cc @Evangelink

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with tests/TestRunner.Core/TestInstrumentation.cs, then inspect the two listed APK project files and the NativeAOT configuration and run the documented NativeAOT command. Compare the existing NUnit discovery, filtering, execution, and TRX paths with MSTest while preserving the listed instrumentation and CI lanes; done means both APK projects and TestRunner.Core no longer use NUnit and all acceptance checks pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
android, csharp
Domain
build-system, ci-cd, mobile-dev, testing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.