android / android/android-test

ATO: Misleading error when `targetInstrumentation` doesn't match `instrumentation` `android:name` in test APK manifest

Open
#1,641 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
1.2k
Forks
342
Avg merge
11h 29m
Merged PRs (30d)
2

Description

### Description

Ideally, allow running with alternate test runner that is different than the test runner listed in the test APK manifest under `instrumentation` `android:name` when using Android Test Orchestrator.

or alternatively, at a minimum, give a better error message.

Currently there is a misleading error message that doesn't tell you what the actual error is.

### Steps to Reproduce

1. create a custom test runner `example.FooRunner extends androidx.test.runner.AndroidJUnitRunner`
2. set `testInstrumentationRunner = "example.FooRunner"`
3. install Android test orchestrator on the device/emulator
4. install app APK and test APK.
5. run this command that has a different test runner/instrumentation ( `AndroidJUnitRunner` instead of `FooRunner` ):
```
CLASSPATH=$(pm path androidx.test.services) app_process / androidx.test.services.shellexecutor.ShellMain \
am instrument -w -e clearPackageData true \
-e targetInstrumentation com.example.test/androidx.test.runner.AndroidJUnitRunner \
-e class androidx.test.orchestrator/.AndroidTestOrchestrator
```

### Expected Results

This should execute using `androidx.test.runner.AndroidJUnitRunner` even though it doesn't match the manifest of the Test APK.

Or at a minimum, give a better warning message in logcat, something like:
```
WARNING: running with Instrumentation (i.e. test runner) that does not match the test APK manifest.
test APK manifest has `instrumentation=`example.FooRunner`,
but using `targetInstrumentation=com.example.test/androidx.test.runner.AndroidJUnitRunner`
```

### Actual Results

Misleading error in logcat is:
```
16:57:28.530 E FATAL EXCEPTION: AndroidTestOrchestrator
Process: androidx.test.orchestrator, PID: 4201
java.lang.IllegalStateException: Package [com.example.test] cannot be found on the system.
at androidx.test.orchestrator.AndroidTestOrchestrator.getTargetPackage(AndroidTestOrchestrator.java:7)
at androidx.test.orchestrator.AndroidTestOrchestrator.-$$Nest$mgetTargetPackage(Unknown Source:0)
at androidx.test.orchestrator.AndroidTestOrchestrator$2.run(AndroidTestOrchestrator.java:4)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:923)
```

`com.example.test` IS installed, but the issue is the mismatched instrumentation class (i.e. test runner)
`Package [com.example.test] cannot be found on the system.` is not correct.

Related code:
https://github.com/android/android-test/blob/13fcb55c1c4ca8156caa61fd20f7e08332a06c04/runner/android_test_orchestrator/java/androidx/test/orchestrator/AndroidTestOrchestrator.java#L525-L538

The instrumentation is use is not really even relevant for that method.
could do something like this instead:

```java
private String getTargetPackage(Bundle arguments) {
String instrPackage = getTargetInstrPackage(arguments);
PackageManager packageManager = getContext().getPackageManager();

try {
InstrumentationInfo instrInfo =
packageManager.getPackageInfo(instrPackage, PackageManager.GET_INSTRUMENTATION)
.instrumentation[0];
return instrInfo.targetPackage;
} catch (NameNotFoundException e) {
throw new IllegalStateException(
"Package [" + instrPackage + "] cannot be found on the system.");
}
}
```

### AndroidX Test and Android OS Versions
- `androidx.test:monitor:1.6.0`
- `androidx.test:runner:1.5.0`
- Android Test Orchestrator `1.4.2`
- API 30 device

Contributor guide

Open the contributing guide

Research direction

Start with runner/android_test_orchestrator/java/androidx/test/orchestrator/AndroidTestOrchestrator.java around lines 525-538 and reproduce the issue with the provided ShellMain command on an API 30 device. Trace how targetInstrumentation and the test APK manifest are resolved, then make the resulting logcat diagnostic identify the mismatched instrumentation instead of reporting the installed package as missing.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
mobile, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.