DTStack / DTStack/chunjun

[issue][logminer and hive] logmier to hive task fail with checkpoint

Open
#1,424 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
4.1k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/DTStack/chunjun/issues) and found no similar question.

- [X] I had googled my question but i didn't get any help.

- [X] I had read the documentation: [ChunJun doc](https://dtstack.github.io/chunjun) but it didn't help me.

### Description

when I close checkpoint , the task runs well whether or not oracle genrate data.

but, when I open cp, the task runs fail cause exception as below and checkpoint fails either causes timeout:
2022-12-02 16:47:36,787 WARN com.dtstack.chunjun.connector.oraclelogminer.listener.LogMinerListener [] - Get data from queue error:
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_181]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) ~[?:1.8.0_181]
at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522) ~[?:1.8.0_181]
at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684) ~[?:1.8.0_181]
at com.dtstack.chunjun.connector.oraclelogminer.listener.LogMinerListener.getData(LogMinerListener.java:248) [chunjun-connector-oraclelogminer.jar:?]
at com.dtstack.chunjun.connector.oraclelogminer.inputformat.OracleLogMinerInputFormat.nextRecordInternal(OracleLogMinerInputFormat.java:92) [chunjun-connector-oraclelogminer.jar:?]
at com.dtstack.chunjun.source.format.BaseRichInputFormat.nextRecord(BaseRichInputFormat.java:198) [chunjun-core.jar:?]
at com.dtstack.chunjun.source.format.BaseRichInputFormat.nextRecord(BaseRichInputFormat.java:68) [chunjun-core.jar:?]
at com.dtstack.chunjun.source.DtInputFormatSourceFunction.run(DtInputFormatSourceFunction.java:133) [chunjun-core.jar:?]
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110) [chunjun-ddl-oracle.jar:?]
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66) [chunjun-ddl-oracle.jar:?]
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:267) [chunjun-ddl-oracle.jar:?]
2022-12-02 16:47:36,795 WARN com.dtstack.chunjun.connector.oraclelogminer.listener.LogMinerConnection [] - close logMiner failed, e = java.sql.SQLException: Closed Statement
at oracle.jdbc.driver.OracleClosedStatement.execute(OracleClosedStatement.java:2365)
at oracle.jdbc.driver.OracleStatementWrapper.execute(OracleStatementWrapper.java:300)
at com.dtstack.chunjun.connector.oraclelogminer.listener.LogMinerConnection.disConnect(LogMinerConnection.java:228)
at java.lang.Iterable.forEach(Iterable.java:75)
at com.dtstack.chunjun.connector.oraclelogminer.listener.LogMinerHelper.stop(LogMinerHelper.java:363)
at com.dtstack.chunjun.connector.oraclelogminer.listener.LogMinerListener.stop(LogMinerListener.java:223)
at com.dtstack.chunjun.connector.oraclelogminer.inputformat.OracleLogMinerInputFormat.closeInternal(OracleLogMinerInputFormat.java:104)
at com.dtstack.chunjun.source.format.BaseRichInputFormat.close(BaseRichInputFormat.java:217)
at com.dtstack.chunjun.source.DtInputFormatSourceFunction.gracefulClose(DtInputFormatSourceFunction.java:170)
at com.dtstack.chunjun.source.DtInputFormatSourceFunction.run(DtInputFormatSourceFunction.java:151)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:267)

2022-12-02 16:47:43,687 WARN com.dtstack.chunjun.connector.oraclelogminer.listener.LogMinerListener [] - LogMinerListener thread exception: current scn =31386935,
e = java.lang.RuntimeException: java.sql.SQLRecoverableException: Closed Connection
at com.dtstack.chunjun.connector.oraclelogminer.listener.LogMinerConnection.startOrUpdateLogMiner(LogMinerConnection.java:296)
at com.dtstack.chunjun.connector.oraclelogminer.listener.LogMinerHelper.preLoad(LogMinerHelper.java:171)
at com.dtstack.chunjun.connector.oraclelogminer.listener.LogMinerHelper.chooseAndPreLoadConnection(LogMinerHelper.java:326)
at com.dtstack.chunjun.connector.oraclelogminer.listener.LogMinerHelper.hasNext(LogMinerHelper.java:294)
at com.dtstack.chunjun.connector.oraclelogminer.listener.LogMinerListener.run(LogMinerListener.java:181)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLRecoverableException: Closed Connection
at oracle.jdbc.driver.PhysicalConnection.prepareCall(PhysicalConnection.java:1708)
at oracle.jdbc.driver.PhysicalConnection.prepareCall(PhysicalConnection.java:1677)
at com.dtstack.chunjun.connector.oraclelogminer.listener.LogMinerConnection.resetLogminerStmt(LogMinerConnection.java:1109)
at com.dtstack.chunjun.connector.oraclelogminer.listener.LogMinerConnection.startOrUpdateLogMiner(LogMinerConnection.java:276)
... 7 more

2022-12-02 16:47:45,687 INFO com.dtstack.chunjun.connector.oraclelogminer.listener.LogMinerHelper [] - restart connection, startScn: 31386936,endScn: 31389678
2022-12-02 16:47:45,688 WARN com.dtstack.chunjun.connector.oraclelogminer.listener.LogMinerConnection [] - close logMiner failed, e = java.sql.SQLException: Closed Statement
at oracle.jdbc.driver.OracleClosedStatement.execute(OracleClosedStatement.java:2365)
at oracle.jdbc.driver.OracleStatementWrapper.execute(OracleStatementWrapper.java:300)
at com.dtstack.chunjun.connector.oraclelogminer.listener.LogMinerConnection.disConnect(LogMinerConnection.java:228)
at com.dtstack.chunjun.connector.oraclelogminer.listener.LogMinerHelper.restart(LogMinerHelper.java:229)
at com.dtstack.chunjun.connector.oraclelogminer.listener.LogMinerHelper.restart(LogMinerHelper.java:219)
at com.dtstack.chunjun.connector.oraclelogminer.listener.LogMinerListener.run(LogMinerListener.java:187)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

============
I learn the code and find where the program throws the exception:
closeStmt(logMinerStartStmt);
logMinerStartStmt = connection.prepareCall(startSql); // execute here then throw exception: closed [connection[

And I checked the exception time and time of relative failed cp ,I found that:
the time of exception is 47:36 while the start time of checkpoint is 47:01 and the cp timeout at 47:36 after 35s that is the value of execution.checkpointing.timeout.

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.