element-hq / element-hq/element-android

Investigate integration test failures when all tests run together

Open
#5,459 2 comments 0 reactions 0 assignees View on GitHub
T-Task X-DevOps z-CI
Dominant language
Kotlin
Stars
3.7k
Forks
917
Avg merge
8d 20h
Merged PRs (30d)
1

Description

```
Mar 08, 2022 2:26:04 PM com.google.testing.platform.RunnerImpl$Companion summarizeAndLogTestResult
SEVERE: Execute org.matrix.android.sdk.internal.crypto.keysbackup.KeysBackupTest.testBackupWhenAnotherBackupWasCreated: FAILED
java.lang.AssertionError: java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at org.matrix.android.sdk.internal.crypto.keysbackup.KeysBackupTest.testBackupWhenAnotherBackupWasCreated(KeysBackupTest.kt:981)
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at org.matrix.android.sdk.internal.crypto.keysbackup.KeysBackupTest.testBackupWhenAnotherBackupWasCreated(KeysBackupTest.kt:981)

Mar 08, 2022 2:26:04 PM com.google.testing.platform.RunnerImpl$Companion summarizeAndLogTestResult
SEVERE: Execute org.matrix.android.sdk.session.space.SpaceHierarchyTest.testParentRelation: FAILED
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java: java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at org.matrix.android.sdk.common.CommonTestHelper.await(CommonTestHelper.kt:378)
at org.matrix.android.sdk.common.CommonTestHelper.waitWithLatch(CommonTestHelper.kt:396)
at org.matrix.android.sdk.common.CommonTestHelper.waitWithLatch$default(CommonTestHelper.kt:391)
at org.matrix.android.sdk.session.space.SpaceHierarchyTest.createPrivateSpace(SpaceHierarchyTest.kt:410)
at org.matrix.android.sdk.session.space.SpaceHierarchyTest.testParentRelation(SpaceHierarchyTest.kt:491)
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at org.matrix.android.sdk.common.CommonTestHelper.await(CommonTestHelper.kt:378)
at org.matrix.android.sdk.common.CommonTestHelper.waitWithLatch(CommonTestHelper.kt:396)
at org.matrix.android.sdk.common.CommonTestHelper.waitWithLatch$default(CommonTestHelper.kt:391)
at org.matrix.android.sdk.session.space.SpaceHierarchyTest.createPrivateSpace(SpaceHierarchyTest.kt:410)
at org.matrix.android.sdk.session.space.SpaceHierarchyTest.testParentRelation(SpaceHierarchyTest.kt:491)

Mar 08, 2022 2:26:04 PM com.google.testing.platform.RunnerImpl$Companion summarizeAndLogTestResult
SEVERE: Execute org.matrix.android.sdk.session.space.SpaceHierarchyTest.testRootSpaces: FAILED
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java: java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at org.matrix.android.sdk.common.CommonTestHelper.await(CommonTestHelper.kt:378)
at org.matrix.android.sdk.common.CommonTestHelper.waitWithLatch(CommonTestHelper.kt:396)
at org.matrix.android.sdk.common.CommonTestHelper.waitWithLatch$default(CommonTestHelper.kt:391)
at org.matrix.android.sdk.session.space.SpaceHierarchyTest.createPublicSpace(SpaceHierarchyTest.kt:382)
at org.matrix.android.sdk.session.space.SpaceHierarchyTest.testRootSpaces(SpaceHierarchyTest.kt:445)
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at org.matrix.android.sdk.common.CommonTestHelper.await(CommonTestHelper.kt:378)
at org.matrix.android.sdk.common.CommonTestHelper.waitWithLatch(CommonTestHelper.kt:396)
at org.matrix.android.sdk.common.CommonTestHelper.waitWithLatch$default(CommonTestHelper.kt:391)
at org.matrix.android.sdk.session.space.SpaceHierarchyTest.createPublicSpace(SpaceHierarchyTest.kt:382)
at org.matrix.android.sdk.session.space.SpaceHierarchyTest.testRootSpaces(SpaceHierarchyTest.kt:445)
```

The above three failures occurred when running `:matrix-android-sdk:connectedDebugAndroidTest` with no package/class limits, in both CI and locally on my desktop.

The tests did not individually fail when running in five separate package-scoped limited runs, so there is some interaction between them (or perhaps between a previous test and them) that is causing this issue.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.