otter往es数据同步出现问题
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 2.5k
- PR merge metrics
- No merged PRs in 30d
Description
之前运行一直正常,现在大量报这个异常,中途接手别人的工作,看不明白哪里出问题了。。。
2021-04-09 15:17:15.726 [pipelineId = 16,taskName = LoadWorker] INFO com.alibaba.otter.node.etl.load.LoadTask - [16] loadWork executor is interrrupt! data:EtlEventData{currNid=18, nextNid=18, desc=[MemoryPipeKey[identity=Identity[channelId=10,pipelineId=16,processId=2],time=1617951917481,dataType=DB_BATCH]], isFull=false}
com.alibaba.otter.node.etl.load.exception.LoadException: java.lang.InterruptedException
Caused by: java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[na:1.8.0_172]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[na:1.8.0_172]
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) ~[na:1.8.0_172]
at java.util.concurrent.ExecutorCompletionService.take(ExecutorCompletionService.java:193) ~[na:1.8.0_172]
at com.alibaba.otter.node.etl.load.loader.db.DataBatchLoader.load(DataBatchLoader.java:109) ~[node.etl-4.2.15-SNAPSHOT.jar:na]
at com.alibaba.otter.node.etl.load.loader.OtterLoaderFactory.load(OtterLoaderFactory.java:48) ~[node.etl-4.2.15-SNAPSHOT.jar:na]
at com.alibaba.otter.node.etl.load.LoadTask$1.run(LoadTask.java:85) ~[node.etl-4.2.15-SNAPSHOT.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_172]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_172]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_172]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_172]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_172]
2021-04-09 15:17:15.726 [Otter-Seda-Executor-47] WARN com.alibaba.otter.node.etl.load.loader.db.ESLoadAction - ##load phase one failed!
java.lang.InterruptedException: null
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404) [na:1.8.0_172]
at java.util.concurrent.FutureTask.get(FutureTask.java:191) [na:1.8.0_172]
at com.alibaba.otter.node.etl.load.loader.db.ESLoadAction.doTwoPhase(ESLoadAction.java:259) [node.etl-4.2.15-SNAPSHOT.jar:na]
at com.alibaba.otter.node.etl.load.loader.db.ESLoadAction.doLoad(ESLoadAction.java:238) [node.etl-4.2.15-SNAPSHOT.jar:na]
at com.alibaba.otter.node.etl.load.loader.db.ESLoadAction.load(ESLoadAction.java:142) [node.etl-4.2.15-SNAPSHOT.jar:na]
--
2021-04-09 15:29:17.046 [Otter-Seda-Executor-56] WARN com.alibaba.otter.node.etl.load.loader.db.ESLoadAction - ##load phase one failed!
java.lang.InterruptedException: null
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404) [na:1.8.0_172]
at java.util.concurrent.FutureTask.get(FutureTask.java:191) [na:1.8.0_172]
at com.alibaba.otter.node.etl.load.loader.db.ESLoadAction.doTwoPhase(ESLoadAction.java:259) [node.etl-4.2.15-SNAPSHOT.jar:na]
at com.alibaba.otter.node.etl.load.loader.db.ESLoadAction.doLoad(ESLoadAction.java:238) [node.etl-4.2.15-SNAPSHOT.jar:na]
at com.alibaba.otter.node.etl.load.loader.db.ESLoadAction.load(ESLoadAction.java:142) [node.etl-4.2.15-SNAPSHOT.jar:na]
at com.alibaba.otter.node.etl.load.loader.db.ESLoadAction$$FastClassBySpringCGLIB$$277dd0db.invoke() [node.etl-4.2.15-SNAPSHOT.jar:na]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core-4.2.8.RELEASE.jar:4.2.8.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:651) [spring-aop-4.2.8.RELEASE.jar:4.2.8.RELEASE]
at com.alibaba.otter.node.etl.load.loader.db.ESLoadAction$$EnhancerBySpringCGLIB$$b6190922.load() [node.etl-4.2.15-SNAPSHOT.jar:na]
at com.alibaba.otter.node.etl.load.loader.db.DataBatchLoader$2.call(DataBatchLoader.java:204) [node.etl-4.2.15-SNAPSHOT.jar:na]
at com.alibaba.otter.node.etl.load.loader.db.DataBatchLoader$2.call(DataBatchLoader.java:194) [node.etl-4.2.15-SNAPSHOT.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_172]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_172]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_172]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_172]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_172]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_172]
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading DataBatchLoader.java around lines 109 and 194-204, then trace the interruption through ESLoadAction.java at lines 142, 238, 259 and the LoadTask stack frames. Determine why the Elasticsearch load phase is interrupted and verify that the pipeline can resume synchronization without these exceptions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100