Don't try to recover Oracle connector in case of archive log absent [DBZ-6220]
- Dominant language
- HTML
- Stars
- 6
- Forks
- 9
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from [DBZ-6220](https://issues.redhat.com/browse/DBZ-6220)
In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.
h1. Bug report
For bug reports, provide this information, please:
h2. What Debezium connector do you use and what version?
Debezium Oracle connector 1.9.5-Final
h2. What is the connector configuration?
No customizations for retriable exceptions, seems the rest doesn't matter in this case but may be provided if necessary.
h2. What is the captured database version and mode of depoyment?
Oracle Database 12c Standard Edition Release 12.2.0.1.0 - 64bit Production
h2. What behaviour do you expect?
In case of the archive log is absent Debezium connector fails.
h2. What behaviour do you see?
The debezium connector restarts in a loop for days. _/status_ endpoint shows RUNNING. Logs're populated with misleading "{_}Couldn't commit processed log positions"{_} (see [this discussion|https://debezium.zulipchat.com/#narrow/stream/302529-users/topic/Couldn't.20commit.20processed.20log.20positions.20with.20the.20source.20data])
h2. Do you see the same behaviour using the latest relesead Debezium version?
Didn't checked
h2. Do you have the connector logs, ideally from start till finish?
(You might be asked later to provide [DEBUG/TRACE|https://debezium.io/documentation/reference/stable/operations/logging.html] level log)
Exception itself
{code:json}
{
"exception": {
"stacktrace": "java.sql.SQLException: ORA-01284: file \/var\/opt\/.../MY_DB\/archive\/1_116358_1111587716.dbf cannot be opened
ORA-00308: cannot open archived log '\/var\/opt\/.../MY_DB\/archive\/1_116358_1111587716.dbf'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7
ORA-06512: at \"SYS.DBMS_LOGMNR\", line 68
ORA-06512: at line 1
at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:628)
...
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1034)
at oracle.jdbc.driver.OracleStatement.executeSQLStatement(OracleStatement.java:1507)
...
at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1010)
at io.debezium.connector.oracle.logminer.LogMinerHelper.executeCallableStatement(LogMinerHelper.java:217)
at io.debezium.connector.oracle.logminer.LogMinerHelper.setLogFilesForMining(LogMinerHelper.java:87)
at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.initializeRedoLogsForMining(LogMinerStreamingChangeEventSource.java:335)
at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:155)
at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:60)
at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:174)
at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:141)
at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:109)
at java.base\/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base\/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base\/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base\/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base\/java.lang.Thread.run(Thread.java:829)
Caused by: Error : 1284, Position : 0, Sql = BEGIN sys.dbms_logmnr.add_logfile(LOGFILENAME => '\/var\/opt/.../MY_DB\/archive\/1_116358_1111587716.dbf', OPTIONS => DBMS_LOGMNR.ADDFILE);END;, OriginalSql = BEGIN sys.dbms_logmnr.add_logfile(LOGFILENAME => '\/var\/opt\.../MY_DB\/archive\/1_116358_1111587716.dbf', OPTIONS => DBMS_LOGMNR.ADDFILE);END;, Error Msg = ORA-01284: file \/var\/opt/.../MY_DB\/archive\/1_116358_1111587716.dbf cannot be opened
ORA-00308: cannot open archived log '\/var\/opt/.../MY_DB\/archive\/1_116358_1111587716.dbf'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7
ORA-06512: at \"SYS.DBMS_LOGMNR\", line 68
ORA-06512: at line 1
at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:632)
... 26 more",
"exception_class": "java.sql.SQLException",
"exception_message": "ORA-01284: file \/var\/opt/.../MY_DB\/archive\/1_116358_1111587716.dbf cannot be opened
ORA-00308: cannot open archived log '\/var\/opt/.../MY_DB\/archive\/1_116358_1111587716.dbf'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7
ORA-06512: at \"SYS.DBMS_LOGMNR\", line 68
ORA-06512: at line 1
"
},
"source_host": "cmd-debezium-kafka-connect-6f56b6cdb9-dbt5c",
"method": "logError",
"level": "ERROR",
"message": "Mining session stopped due to the {}",
"mdc": {
"dbz.connectorContext": "streaming",
"dbz.connectorType": "Oracle",
"connector.context": "[oracle-customers-connector|task-0] ",
"dbz.taskId": "0",
"dbz.connectorName": "my-db"
},
"@timestamp": "2023-03-14T19:42:12.790Z",
"file": "LogMinerHelper.java",
"line_number": "117",
"thread_name": "debezium-oracleconnector-my-db-change-event-source-coordinator",
"@version": 1,
"logger_name": "io.debezium.connector.oracle.logminer.LogMinerHelper",
"class": "io.debezium.connector.oracle.logminer.LogMinerHelper"
}{code}
Causes repeatable restarts
!image-2023-03-17-19-58-58-784.png|width=600!
Full logs and thread dumps may be provided if the issue is not clear from the above snippets
h2. How to reproduce the issue using our [tutorial|https://github.com/debezium/debezium-examples/tree/main/tutorial] deployment?
Remove the archive log while the connector is running
h1. Feature request or enhancement
For feature requests or enhancements, provide this information, please:
h2. Which use case/requirement will be addressed by the proposed feature?
* Timely and easy detection of the issue - now it could be detected only by alerting on the absence of messages in Kafka topic or errors in logs.
* Convenient troubleshooting - no misleading "{_}Couldn't commit processed log positions{_}" caused by livelock.
h2. Implementation ideas (optional)
Do not use retriable exception for this case in ErrorHandler
Contributor guide
Assessment
This issue has not been assessed yet.