android / android/android-test

Frequent 'No tests found' when sharding tests

Open
#973 5 comments 2 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
1.2k
Forks
342
Avg merge
11h 29m
Merged PRs (30d)
2

Description

### Description

When sharding tests I get frequent (and seemingly random) 'No tests found' errors which fail the build.

See this PR: https://github.com/google/accompanist/pull/463 for an example.

### Steps to Reproduce

I'm not 100% sure what the repro is. Having a small number of tests in the module seems to be the trigger.

I think this is because the library is relying on [hashCode being uniformly random](https://github.com/android/android-test/blob/9d25d24ec7abe5021672c0423a992e87b44eb261/runner/android_junit_runner/java/androidx/test/internal/runner/TestRequestBuilder.java#L338) over a small data-set, which isn't the case. If the module only has ~5 tests, and I'm sharding over 3 runs, the chance of at least 1 shard having 0 tests is quite high.

### Expected Results

Shards with no tests from a module don't fail the build.

### Actual Results

```
com.android.build.gradle.internal.testing.ConnectedDevice > No tests found.[test(AVD) - 5.1.1] FAILED
No tests found. This usually means that your test classes are not in the form that your test runner expects (e.g. don't inherit from TestCase or lack @Test annotations).
```

### AndroidX Test and Android OS Versions

[`1.3.0`](https://github.com/google/accompanist/blob/main/gradle/libs.versions.toml#L11)

### Link to a public git repo demonstrating the problem:

https://github.com/google/accompanist/pull/463

Contributor guide

Open the contributing guide

Research direction

Start with the TestRequestBuilder.java implementation linked in the issue and the sharding behavior demonstrated by accompanist PR 463. Reproduce the small-module, multi-shard case and verify that shards receiving no tests no longer fail the build.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
mobile-dev, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.