android / android/android-test
ATO: Misleading error when `targetInstrumentation` doesn't match `instrumentation` `android:name` in test APK manifest
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 342
- Avg merge
- 11h 29m
- Merged PRs (30d)
- 2
Description
### Description
Ideally, allow running with alternate test runner that is different than the test runner listed in the test APK manifest under `instrumentation` `android:name` when using Android Test Orchestrator.
or alternatively, at a minimum, give a better error message.
Currently there is a misleading error message that doesn't tell you what the actual error is.
### Steps to Reproduce
1. create a custom test runner `example.FooRunner extends androidx.test.runner.AndroidJUnitRunner`
2. set `testInstrumentationRunner = "example.FooRunner"`
3. install Android test orchestrator on the device/emulator
4. install app APK and test APK.
5. run this command that has a different test runner/instrumentation ( `AndroidJUnitRunner` instead of `FooRunner` ):
```
CLASSPATH=$(pm path androidx.test.services) app_process / androidx.test.services.shellexecutor.ShellMain \
am instrument -w -e clearPackageData true \
-e targetInstrumentation com.example.test/androidx.test.runner.AndroidJUnitRunner \
-e class androidx.test.orchestrator/.AndroidTestOrchestrator
```
### Expected Results
This should execute using `androidx.test.runner.AndroidJUnitRunner` even though it doesn't match the manifest of the Test APK.
Or at a minimum, give a better warning message in logcat, something like:
```
WARNING: running with Instrumentation (i.e. test runner) that does not match the test APK manifest.
test APK manifest has `instrumentation=`example.FooRunner`,
but using `targetInstrumentation=com.example.test/androidx.test.runner.AndroidJUnitRunner`
```
### Actual Results
Misleading error in logcat is:
```
16:57:28.530 E FATAL EXCEPTION: AndroidTestOrchestrator
Process: androidx.test.orchestrator, PID: 4201
java.lang.IllegalStateException: Package [com.example.test] cannot be found on the system.
at androidx.test.orchestrator.AndroidTestOrchestrator.getTargetPackage(AndroidTestOrchestrator.java:7)
at androidx.test.orchestrator.AndroidTestOrchestrator.-$$Nest$mgetTargetPackage(Unknown Source:0)
at androidx.test.orchestrator.AndroidTestOrchestrator$2.run(AndroidTestOrchestrator.java:4)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:923)
```
`com.example.test` IS installed, but the issue is the mismatched instrumentation class (i.e. test runner)
`Package [com.example.test] cannot be found on the system.` is not correct.
Related code:
https://github.com/android/android-test/blob/13fcb55c1c4ca8156caa61fd20f7e08332a06c04/runner/android_test_orchestrator/java/androidx/test/orchestrator/AndroidTestOrchestrator.java#L525-L538
The instrumentation is use is not really even relevant for that method.
could do something like this instead:
```java
private String getTargetPackage(Bundle arguments) {
String instrPackage = getTargetInstrPackage(arguments);
PackageManager packageManager = getContext().getPackageManager();
try {
InstrumentationInfo instrInfo =
packageManager.getPackageInfo(instrPackage, PackageManager.GET_INSTRUMENTATION)
.instrumentation[0];
return instrInfo.targetPackage;
} catch (NameNotFoundException e) {
throw new IllegalStateException(
"Package [" + instrPackage + "] cannot be found on the system.");
}
}
```
### AndroidX Test and Android OS Versions
- `androidx.test:monitor:1.6.0`
- `androidx.test:runner:1.5.0`
- Android Test Orchestrator `1.4.2`
- API 30 device
Contributor guide
Research direction
Start with runner/android_test_orchestrator/java/androidx/test/orchestrator/AndroidTestOrchestrator.java around lines 525-538 and reproduce the issue with the provided ShellMain command on an API 30 device. Trace how targetInstrumentation and the test APK manifest are resolved, then make the resulting logcat diagnostic identify the mismatched instrumentation instead of reporting the installed package as missing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- mobile, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100