android / android/android-test

ClassNotFoundException: no tests found on Orchestrator sharded runs

Open
#264 3 comments 12 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
1.2k
Forks
342
Avg merge
11h 29m
Merged PRs (30d)
2

Description

When I'm running sharded instrumentation tests with Orchestrator from the command line I encounter nondeterministic behavior which strongly depends on testing package name (or class name maybe?).
One version works fine, tests from the package are taken one after another. In another version I encounter errors:

```
java.lang.ClassNotFoundException: Invalid name: no tests found
at java.lang.Class.classForName(Native Method)
at java.lang.Class.forName(Class.java:453)
at androidx.test.internal.runner.TestLoader.doCreateRunner(TestLoader.java:72)
at androidx.test.internal.runner.TestLoader.getRunnersFor(TestLoader.java:104)
at androidx.test.internal.runner.TestRequestBuilder.build(TestRequestBuilder.java:789)
at androidx.test.runner.AndroidJUnitRunner.buildRequest(AndroidJUnitRunner.java:543)
at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:386)
at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2145)
```

It always happens for shardIndex=0 (shardsNum=2 or 3).

It’s not the cache probably. We can reproduce this issue on different machines.

Command
```
adb -s emulator-5554 shell 'CLASSPATH=$(pm path androidx.test.services) app_process / androidx.test.services.shellexecutor.ShellMain am instrument -w -r -e targetInstrumentation com.azimo.sendmoney.debug1.test/com.azimo.sendmoney.instrumentation.config.AzimoTestRunner -e package com.azimo.sendmoney.instrumentation.azimoTestCases.functional.AccountVerification -e numShards 2 -e shardIndex 0 -e clearPackageData true androidx.test.orchestrator/.AndroidTestOrchestrator'
```

When I change package name by single letter:
From: `com.azimo.sendmoney.debug1.test/com.azimo.sendmoney.instrumentation.config.AzimoTestRunner`
To (“A” at the end): `com.azimo.sendmoney.debug1.test/com.azimo.sendmoney.instrumentation.config.AzimoTestRunnerA`

Problems don't happen.

Also if I run package in two shards, index=0 and index=1, despite java.lang.ClassNotFoundException error, all tests are run correctly (but they are all run on the emulator running shardIndex=1).

Versions
Test runner: 1.1.1
Espresso: 3.1.1
Orchestrator: 1.1.1 apk
Test services: 1.1.1 apk
Proguard - tried with minify enabled and disabled. Same results.

Our test runner does nothing unusual.
```
public class AzimoTestRunner extends AndroidJUnitRunner {

@Override
public Application newApplication(ClassLoader cl, String className, Context context)
throws InstantiationException, IllegalAccessException, ClassNotFoundException {
return super.newApplication(cl, AzimoTestApplication.class.getName(), context);
}
}
```
Interesting fact. When I log values from onCreate method bundle, failing run gets:

shardIndex 0
clearPackageData true
shellExecKey cb88bd77f522eeeb
package com.azimo.sendmoney.instrumentation.azimoTestCases.functional.AccountVerification
listTestsForOrchestrator true
numShards 2
orchestratorService OrchestratorService

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the command-line Orchestrator run with shardIndex 0, then trace the failure through TestLoader.java, TestRequestBuilder.java, and AndroidJUnitRunner.java as shown in the stack trace. Compare shardIndex 0 and 1 with the listed 1.1.1 versions; done means sharded runs no longer report ClassNotFoundException and execute tests on the intended shard.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
mobile, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.