apache / apache/druid

Flaky integration test for input sources

Open
#11,744 2 comments 0 reactions 0 assignees View on GitHub
Development Blocker Flaky test
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

There are 3 jobs for integration testing for input sources running on Travis. These tests seem to be almost always failing recently. The stack trace below is copied from https://app.travis-ci.com/github/apache/druid/builds/238435945, which is commonly found in those failed jobs.

```
2021-09-27T18:06:17,485 ERROR [main] org.apache.druid.testing.utils.LoggerListener - Failed [org.apache.druid.tests.indexer.ITHttpInputSourceTest.doTest]
org.apache.druid.java.util.common.ISE: Max number of retries[240] exceeded for Task[index_parallel_wikipedia_http_inputsource_test_85c6a767-8acb-4219-9747-189741b98916 %Россия 한국 中国!?_ddhmammk_2021-09-27T17:46:02.898Z]. Failing.
at org.apache.druid.testing.utils.ITRetryUtil.retryUntil(ITRetryUtil.java:94) ~[druid-integration-tests-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
at org.apache.druid.testing.clients.OverlordResourceTestClient.waitUntilTaskCompletes(OverlordResourceTestClient.java:282) ~[druid-integration-tests-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
at org.apache.druid.testing.clients.OverlordResourceTestClient.waitUntilTaskCompletes(OverlordResourceTestClient.java:277) ~[druid-integration-tests-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
at org.apache.druid.tests.indexer.AbstractITBatchIndexTest.submitTaskAndWait(AbstractITBatchIndexTest.java:323) ~[test-classes/:?]
at org.apache.druid.tests.indexer.AbstractITBatchIndexTest.doIndexTest(AbstractITBatchIndexTest.java:146) ~[test-classes/:?]
at org.apache.druid.tests.indexer.AbstractITBatchIndexTest.doIndexTest(AbstractITBatchIndexTest.java:114) ~[test-classes/:?]
at org.apache.druid.tests.indexer.ITHttpInputSourceTest.doTest(ITHttpInputSourceTest.java:44) ~[test-classes/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_252]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_252]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132) ~[testng-7.3.0.jar:?]
at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:599) ~[testng-7.3.0.jar:?]
at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:174) ~[testng-7.3.0.jar:?]
at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46) ~[testng-7.3.0.jar:?]
at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822) ~[testng-7.3.0.jar:?]
at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147) ~[testng-7.3.0.jar:?]
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146) ~[testng-7.3.0.jar:?]
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128) ~[testng-7.3.0.jar:?]
at java.util.ArrayList.forEach(ArrayList.java:1257) [?:1.8.0_252]
at org.testng.TestRunner.privateRun(TestRunner.java:764) [testng-7.3.0.jar:?]
at org.testng.TestRunner.run(TestRunner.java:585) [testng-7.3.0.jar:?]
at org.testng.DruidTestRunnerFactory$DruidTestRunner.runTests(DruidTestRunnerFactory.java:101) [druid-integration-tests-0.23.0-SNAPSHOT.jar:?]
at org.testng.DruidTestRunnerFactory$DruidTestRunner.run(DruidTestRunnerFactory.java:88) [druid-integration-tests-0.23.0-SNAPSHOT.jar:?]
at org.testng.SuiteRunner.runTest(SuiteRunner.java:384) [testng-7.3.0.jar:?]
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378) [testng-7.3.0.jar:?]
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337) [testng-7.3.0.jar:?]
at org.testng.SuiteRunner.run(SuiteRunner.java:286) [testng-7.3.0.jar:?]
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53) [testng-7.3.0.jar:?]
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96) [testng-7.3.0.jar:?]
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218) [testng-7.3.0.jar:?]
at org.testng.TestNG.runSuitesLocally(TestNG.java:1140) [testng-7.3.0.jar:?]
at org.testng.TestNG.runSuites(TestNG.java:1069) [testng-7.3.0.jar:?]
at org.testng.TestNG.run(TestNG.java:1037) [testng-7.3.0.jar:?]
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:283) [surefire-testng-2.22.0.jar:2.22.0]
at org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:75) [surefire-testng-2.22.0.jar:2.22.0]
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:120) [surefire-testng-2.22.0.jar:2.22.0]
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:383) [surefire-booter-2.22.0.jar:2.22.0]
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:344) [surefire-booter-2.22.0.jar:2.22.0]
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125) [surefire-booter-2.22.0.jar:2.22.0]
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:417) [surefire-booter-2.22.0.jar:2.22.0]
Suppressed: java.lang.RuntimeException: java.lang.RuntimeException: org.apache.druid.java.util.common.ISE: Error while making request to url[http://127.0.0.1:8081/druid/coordinator/v1/datasources/wikipedia_http_inputsource_test_85c6a767-8acb-4219-9747-189741b98916%20%25%D0%A0%D0%BE%D1%81%D1%81%D0%B8%D1%8F%20%ED%95%9C%EA%B5%AD%20%E4%B8%AD%E5%9B%BD%21%3F/intervals] status[204 No Content] content[]
at org.apache.druid.testing.clients.CoordinatorResourceTestClient.getSegmentIntervals(CoordinatorResourceTestClient.java:154) ~[druid-integration-tests-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
at org.apache.druid.tests.indexer.AbstractIndexerTest.unloadAndKillData(AbstractIndexerTest.java:91) ~[test-classes/:?]
at org.apache.druid.tests.indexer.AbstractIndexerTest.lambda$unloader$0(AbstractIndexerTest.java:71) ~[test-classes/:?]
at org.apache.druid.tests.indexer.ITHttpInputSourceTest.doTest(ITHttpInputSourceTest.java:53) ~[test-classes/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_252]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_252]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132) ~[testng-7.3.0.jar:?]
at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:599) ~[testng-7.3.0.jar:?]
at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:174) ~[testng-7.3.0.jar:?]
at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46) ~[testng-7.3.0.jar:?]
at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822) ~[testng-7.3.0.jar:?]
at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147) ~[testng-7.3.0.jar:?]
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146) ~[testng-7.3.0.jar:?]
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128) ~[testng-7.3.0.jar:?]
at java.util.ArrayList.forEach(ArrayList.java:1257) [?:1.8.0_252]
at org.testng.TestRunner.privateRun(TestRunner.java:764) [testng-7.3.0.jar:?]
at org.testng.TestRunner.run(TestRunner.java:585) [testng-7.3.0.jar:?]
at org.testng.DruidTestRunnerFactory$DruidTestRunner.runTests(DruidTestRunnerFactory.java:101) [druid-integration-tests-0.23.0-SNAPSHOT.jar:?]
at org.testng.DruidTestRunnerFactory$DruidTestRunner.run(DruidTestRunnerFactory.java:88) [druid-integration-tests-0.23.0-SNAPSHOT.jar:?]
at org.testng.SuiteRunner.runTest(SuiteRunner.java:384) [testng-7.3.0.jar:?]
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378) [testng-7.3.0.jar:?]
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337) [testng-7.3.0.jar:?]
at org.testng.SuiteRunner.run(SuiteRunner.java:286) [testng-7.3.0.jar:?]
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53) [testng-7.3.0.jar:?]
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96) [testng-7.3.0.jar:?]
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218) [testng-7.3.0.jar:?]
at org.testng.TestNG.runSuitesLocally(TestNG.java:1140) [testng-7.3.0.jar:?]
at org.testng.TestNG.runSuites(TestNG.java:1069) [testng-7.3.0.jar:?]
at org.testng.TestNG.run(TestNG.java:1037) [testng-7.3.0.jar:?]
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:283) [surefire-testng-2.22.0.jar:2.22.0]
at org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:75) [surefire-testng-2.22.0.jar:2.22.0]
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:120) [surefire-testng-2.22.0.jar:2.22.0]
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:383) [surefire-booter-2.22.0.jar:2.22.0]
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:344) [surefire-booter-2.22.0.jar:2.22.0]
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125) [surefire-booter-2.22.0.jar:2.22.0]
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:417) [surefire-booter-2.22.0.jar:2.22.0]
Caused by: java.lang.RuntimeException: org.apache.druid.java.util.common.ISE: Error while making request to url[http://127.0.0.1:8081/druid/coordinator/v1/datasources/wikipedia_http_inputsource_test_85c6a767-8acb-4219-9747-189741b98916%20%25%D0%A0%D0%BE%D1%81%D1%81%D0%B8%D1%8F%20%ED%95%9C%EA%B5%AD%20%E4%B8%AD%E5%9B%BD%21%3F/intervals] status[204 No Content] content[]
at org.apache.druid.testing.clients.CoordinatorResourceTestClient.makeRequest(CoordinatorResourceTestClient.java:433) ~[druid-integration-tests-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
at org.apache.druid.testing.clients.CoordinatorResourceTestClient.getSegmentIntervals(CoordinatorResourceTestClient.java:145) ~[druid-integration-tests-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
... 37 more
Caused by: org.apache.druid.java.util.common.ISE: Error while making request to url[http://127.0.0.1:8081/druid/coordinator/v1/datasources/wikipedia_http_inputsource_test_85c6a767-8acb-4219-9747-189741b98916%20%25%D0%A0%D0%BE%D1%81%D1%81%D0%B8%D1%8F%20%ED%95%9C%EA%B5%AD%20%E4%B8%AD%E5%9B%BD%21%3F/intervals] status[204 No Content] content[]
at org.apache.druid.testing.clients.CoordinatorResourceTestClient.makeRequest(CoordinatorResourceTestClient.java:427) ~[druid-integration-tests-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
at org.apache.druid.testing.clients.CoordinatorResourceTestClient.getSegmentIntervals(CoordinatorResourceTestClient.java:145) ~[druid-integration-tests-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
... 37 more
```

Contributor guide

Open the contributing guide

Research direction

Start with ITHttpInputSourceTest.doTest and the stack-trace path through AbstractITBatchIndexTest, AbstractIndexerTest, and CoordinatorResourceTestClient. Reproduce the failing input-source integration jobs from the linked Travis build, then trace the retry and 204 response behavior. Done means the three Travis jobs run reliably without this failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.