android / android/android-test

Instrumentation did not complete failure when no tests are scheduled to run

Open
#2,265 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
1.2k
Forks
342
Avg merge
11h 29m
Merged PRs (30d)
2

Description

### Description
I have stumbled upon an issue where, if my instrumentation test filter is not able to run any tests, gradle build fails. Is there a way to avoid this ?

I have a test suite consisting of tablet and phone tests, and during runtime I am doing custom filtering based on the characteristic of device under test using [CustomFilter](https://developer.android.com/reference/androidx/test/filters/CustomFilter) API's. So, I have device pool consisting of a phone and tablet device. And I have having a class level annotation that specify tests to only target the tablet devices. As expected, all the tests run on tablet device. Since, there were no tests scheduled for the phone device. Even after all the test finished successfully, the build failed with the following error:

```
> Task :app:connectedDebugAndroidTest
Starting 0 tests on Pixel_6_API_34(AVD) - 14

Finished 0 tests on Pixel_6_API_34(AVD) - 14

> Task :app:connectedDebugAndroidTest
Starting 4 tests on Pixel_Tablet_API_34(AVD) - 14

Pixel_Tablet_API_34(AVD) - 14 Tests 1/4 completed. (0 skipped) (0 failed)
Pixel_Tablet_API_34(AVD) - 14 Tests 2/4 completed. (0 skipped) (0 failed)
Pixel_Tablet_API_34(AVD) - 14 Tests 3/4 completed. (0 skipped) (0 failed)
Finished 4 tests on Pixel_Tablet_API_34(AVD) - 14

> Task :app:connectedDebugAndroidTest
Instrumentation did not complete:
Instrumentation code: 0
Instrumentation result bundle:

> Task :app:connectedDebugAndroidTest FAILED
```

```
INSTRUMENTATION_RESULT: stream=

Time: 0

OK (0 tests)

INSTRUMENTATION_CODE: 0
```
this is the `test-results.log` file generated under `app/build/outputs/androidTest-results` dir for Pixel_6 device. I suspect instrumentation code need to be -1 for instrumentation to succeed.

```





Instrumentation did not complete:
Instrumentation code: 0
Instrumentation result bundle:

```
xml report seems to contain `system-err` tag generated for the Pixel_6 device.

### Steps to Reproduce
Spin up an emultor with tablet resolution(landscape) and other one with phone resolution(portrait)
Run test as usually and specify className as instumentation filter args.

### Expected Results
The gradle build should succeed as it run tests on the desired device factor, skipping tests only on the non-desired device.

### Actual Results
even though the test run succeed, the gradle build failed

### AndroidX Test and Android OS Versions
AGP - 8.5.1
Espresso - 3.6.1
androidXTestCore - 1.6.1
androidXTestJUnit - 1.2.1
androidXTestRules - 1.6.1
androidXTestRunner - 1.6.1
androidXTestOrchestrator - 1.5.0

### Link to a public git repo demonstrating the problem:

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with one tablet and one phone emulator using the CustomFilter instrumentation filter, then inspect app/build/outputs/androidTest-results/test-results.log for the zero-test device. Trace how the connectedDebugAndroidTest task handles an instrumentation run with zero scheduled tests; done means the build succeeds when other devices complete their tests and one device schedules none.

Written by the indexing model from the issue text.

Assessment

Tech stack
android, java
Domain
mobile, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.