android / android/android-test
ClassNotFoundException: no tests found on Orchestrator sharded runs
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 342
- Avg merge
- 11h 29m
- Merged PRs (30d)
- 2
Description
When I'm running sharded instrumentation tests with Orchestrator from the command line I encounter nondeterministic behavior which strongly depends on testing package name (or class name maybe?).
One version works fine, tests from the package are taken one after another. In another version I encounter errors:
```
java.lang.ClassNotFoundException: Invalid name: no tests found
at java.lang.Class.classForName(Native Method)
at java.lang.Class.forName(Class.java:453)
at androidx.test.internal.runner.TestLoader.doCreateRunner(TestLoader.java:72)
at androidx.test.internal.runner.TestLoader.getRunnersFor(TestLoader.java:104)
at androidx.test.internal.runner.TestRequestBuilder.build(TestRequestBuilder.java:789)
at androidx.test.runner.AndroidJUnitRunner.buildRequest(AndroidJUnitRunner.java:543)
at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:386)
at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2145)
```
It always happens for shardIndex=0 (shardsNum=2 or 3).
It’s not the cache probably. We can reproduce this issue on different machines.
Command
```
adb -s emulator-5554 shell 'CLASSPATH=$(pm path androidx.test.services) app_process / androidx.test.services.shellexecutor.ShellMain am instrument -w -r -e targetInstrumentation com.azimo.sendmoney.debug1.test/com.azimo.sendmoney.instrumentation.config.AzimoTestRunner -e package com.azimo.sendmoney.instrumentation.azimoTestCases.functional.AccountVerification -e numShards 2 -e shardIndex 0 -e clearPackageData true androidx.test.orchestrator/.AndroidTestOrchestrator'
```
When I change package name by single letter:
From: `com.azimo.sendmoney.debug1.test/com.azimo.sendmoney.instrumentation.config.AzimoTestRunner`
To (“A” at the end): `com.azimo.sendmoney.debug1.test/com.azimo.sendmoney.instrumentation.config.AzimoTestRunnerA`
Problems don't happen.
Also if I run package in two shards, index=0 and index=1, despite java.lang.ClassNotFoundException error, all tests are run correctly (but they are all run on the emulator running shardIndex=1).
Versions
Test runner: 1.1.1
Espresso: 3.1.1
Orchestrator: 1.1.1 apk
Test services: 1.1.1 apk
Proguard - tried with minify enabled and disabled. Same results.
Our test runner does nothing unusual.
```
public class AzimoTestRunner extends AndroidJUnitRunner {
@Override
public Application newApplication(ClassLoader cl, String className, Context context)
throws InstantiationException, IllegalAccessException, ClassNotFoundException {
return super.newApplication(cl, AzimoTestApplication.class.getName(), context);
}
}
```
Interesting fact. When I log values from onCreate method bundle, failing run gets:
shardIndex 0
clearPackageData true
shellExecKey cb88bd77f522eeeb
package com.azimo.sendmoney.instrumentation.azimoTestCases.functional.AccountVerification
listTestsForOrchestrator true
numShards 2
orchestratorService OrchestratorService
Contributor guide
Research direction
Start by reproducing the command-line Orchestrator run with shardIndex 0, then trace the failure through TestLoader.java, TestRequestBuilder.java, and AndroidJUnitRunner.java as shown in the stack trace. Compare shardIndex 0 and 1 with the listed 1.1.1 versions; done means sharded runs no longer report ClassNotFoundException and execute tests on the intended shard.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- mobile, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100