GoogleContainerTools / GoogleContainerTools/jib

Found a flaky test in DockerManifestEntryTemplateTest.java

Open
#4,136 4 comments 0 reactions 0 assignees View on GitHub
priority:p3 type: process
Dominant language
Java
Stars
14.5k
Forks
1.5k
PR merge metrics
No merged PRs in 30d

Description

**Environment**:

- *Jib version:*
- *Build tool:* gradle-6.9.2
- *OS:* MacOS

**Description of the issue**:

Found a flaky test in DockerManifestEntryTemplateTest.testToJson

https://github.com/GoogleContainerTools/jib/blob/440bd4897c6c427cc9a491128aa76c72378d605b/jib-core/src/test/java/com/google/cloud/tools/jib/docker/json/DockerManifestEntryTemplateTest.java#L52

When comparing JSON strings, since JSON is inherently unordered (the order of keys in JSON objects is not guaranteed), it's not reliable to compare JSON strings directly if the order of the keys may vary. This makes this test flaky i.e. non-deterministic. Instead, we should parse the JSON strings into objects and compare the objects semantically.

**Expected behavior**:
We use Jackson's ObjectMapper to convert the JSON strings into JsonNode objects. Then it compares these JsonNode objects with the equals method, which does a deep comparison of the content without considering the order of the keys.
```
ObjectMapper mapper = new ObjectMapper();
JsonNode actual = mapper.readTree(JsonTemplateMapper.toUtf8String(loadManifest));
JsonNode expected = mapper.readTree(expectedJson);
Assert.assertEquals(expected, actual);
```

I've also raised a tentative PR - https://github.com/harshith2000/jib/pull/2

**Steps to reproduce**:

I utilized the open-source tool [NonDex](https://github.com/TestingResearchIllinois/NonDex) to detect this assumption by altering the order of returned exception types.

**To replicate:**
> Clone the Repository:
> ```
> https://github.com/GoogleContainerTools/jib.git
> ```

**Integrate NonDex:**
> Add the following snippet to the top of the build.gradle file:
> ```
> plugins {
> id 'edu.illinois.nondex' version '2.1.1-1'
> }
> ```
> **Add to the end of the build.gradle file:**
> ```
> apply plugin: 'edu.illinois.nondex'
> subprojects {
> apply plugin: 'edu.illinois.nondex'
> }
> ```
**Execute Test with Gradle:**
> ```
> ./gradlew --info test --tests com.google.cloud.tools.jib.docker.json.DockerManifestEntryTemplateTest.testToJson
> ```
**Run NonDex:**
> ```
> ./gradlew --info nondexTest --tests=com.google.cloud.tools.jib.docker.json.DockerManifestEntryTemplateTest.testToJson --nondexRuns=2
> ```

**Log output**:
```
com.google.cloud.tools.jib.docker.json.DockerManifestEntryTemplateTest > testToJson FAILED
org.junit.ComparisonFailure: expected:<[{"[Config":"config.json","RepoTags":["testregistry/testrepo:testtag"],"Layers":["layer1.tar.gz","layer2.tar.gz","layer3.tar.gz"]]}]> but was:<[{"[RepoTags":["testregistry/testrepo:testtag"],"Layers":["layer1.tar.gz","layer2.tar.gz","layer3.tar.gz"],"Config":"config.json"]}]>
at org.junit.Assert.assertEquals(Assert.java:117)
at org.junit.Assert.assertEquals(Assert.java:146)
at com.google.cloud.tools.jib.docker.json.DockerManifestEntryTemplateTest.testToJson(DockerManifestEntryTemplateTest.java:52)

```

Please let me know if you have any questions.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.