google / google/guice

8 Flaky tests in the module extensions/throwingproviders

Open
#1,767 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
12.7k
Forks
1.7k
Avg merge
11m
Merged PRs (30d)
2

Description

### Found 8 flaky tests in the module extensions/throwingproviders

### 1. Tests that fail

### Module: extensions/throwingproviders
> Class Name1: com.google.inject.throwingproviders.CheckedProvidersTest,
> > Test Name: [testUnsupportedMethods_otherMethod_throwsIllegalArgumentException](https://github.com/google/guice/blob/73dc9f3b8afac8b742b766049afc69caf95dbf51/extensions/throwingproviders/test/com/google/inject/throwingproviders/CheckedProvidersTest.java#L78)

> Class Name2: testUnsupportedMethods_otherMethod_throwsIllegalArgumentException
> > Tests Names: [testCxtorWithManyExceptions](https://github.com/google/guice/blob/73dc9f3b8afac8b742b766049afc69caf95dbf51/extensions/throwingproviders/test/com/google/inject/throwingproviders/CheckedProviderTest.java#L1287), [testCxtorWithSuperclassExceptionFails](https://github.com/google/guice/blob/73dc9f3b8afac8b742b766049afc69caf95dbf51/extensions/throwingproviders/test/com/google/inject/throwingproviders/CheckedProviderTest.java#L1143), [testCxtorWithWrongException](https://github.com/google/guice/blob/73dc9f3b8afac8b742b766049afc69caf95dbf51/extensions/throwingproviders/test/com/google/inject/throwingproviders/CheckedProviderTest.java#L1014), [testManyMethods](https://github.com/google/guice/blob/73dc9f3b8afac8b742b766049afc69caf95dbf51/extensions/throwingproviders/test/com/google/inject/throwingproviders/CheckedProviderTest.java#L1413), [testProviderMethodWithManyExceptions](https://github.com/google/guice/blob/73dc9f3b8afac8b742b766049afc69caf95dbf51/extensions/throwingproviders/test/com/google/inject/throwingproviders/CheckedProviderTest.java#L1242), [testProviderMethodWithSuperclassExceptionFails](https://github.com/google/guice/blob/73dc9f3b8afac8b742b766049afc69caf95dbf51/extensions/throwingproviders/test/com/google/inject/throwingproviders/CheckedProviderTest.java#L1113), [testProviderMethodWithWrongException](https://github.com/google/guice/blob/73dc9f3b8afac8b742b766049afc69caf95dbf51/extensions/throwingproviders/test/com/google/inject/throwingproviders/CheckedProviderTest.java#L984)

### 2. Why these tests fail?

The test - testUnsupportedMethods_otherMethod_throwsIllegalArgumentException in CheckedProvidersTest.java has the getDeclaredMethods().
According to the official [documentation](https://docs.oracle.com/javase/8/docs/api/java/lang/Class.html#getDeclaredMethods--), this returns the methods in a different order each time it is called, making this test flaky i.e. non-deterministic.
The tests testCxtorWithManyExceptions, testCxtorWithSuperclassExceptionFails, testCxtorWithWrongException, testManyMethods, testProviderMethodWithManyExceptions, testProviderMethodWithSuperclassExceptionFails, testProviderMethodWithWrongException in CheckedProviderTest.java has the getErrorMessages().
This returns the error messages in a different order each time it is called. For instance, sometimes it returns [RemoteException, BindException], but at other times, it returns [BindException, RemoteException].
Since there is no guarantee in the order of exception types returned, the order might change, resulting in the test failure.

### 3. How to reproduce?

I used an open-source tool called [NonDex](https://github.com/TestingResearchIllinois/NonDex) to detect the assumption by shuffling the order of returned exception types.
Running the following commands will test the aforementioned operation

### Clone the Repo
```
https://github.com/google/guice
```
### Compile the module
```
mvn install extensions/throwingproviders -am -DskipTests
```
### (Optional) Run the unit test
```
mvn -pl throwingproviders test -Dtest=com.google.inject.throwingproviders.CheckedProvidersTest#testUnsupportedMethods_otherMethod_throwsIllegalArgumentException
```
### Run the unit test using NonDex
```
mvn -pl throwingproviders edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=com.google.inject.throwingproviders.CheckedProvidersTest#testUnsupportedMethods_otherMethod_throwsIllegalArgumentException
```
### 4. How I fixed?

Since the order of the methods or error messages is not guaranteed, it is important to consider all possible orders. Fortunately, only two possible orders are possible for both methods and error messages. By checking if the actual message is equal to any of the orders, we can fix the flakiness.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.