debezium / debezium/dbz

Don't try to recover Oracle connector in case of archive log absent [DBZ-6220]

Open
#783 0 comments 0 reactions 1 assignee Claimed by @vjuranek View on GitHub
component/oracle-connector migrated-from-jira type/bug
Dominant language
HTML
Stars
6
Forks
9
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-6220](https://issues.redhat.com/browse/DBZ-6220)

In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.
h1. Bug report

For bug reports, provide this information, please:
h2. What Debezium connector do you use and what version?

Debezium Oracle connector 1.9.5-Final
h2. What is the connector configuration?

No customizations for retriable exceptions, seems the rest doesn't matter in this case but may be provided if necessary.
h2. What is the captured database version and mode of depoyment?

Oracle Database 12c Standard Edition Release 12.2.0.1.0 - 64bit Production
h2. What behaviour do you expect?

In case of the archive log is absent Debezium connector fails.
h2. What behaviour do you see?

The debezium connector restarts in a loop for days. _/status_ endpoint shows RUNNING. Logs're populated with misleading "{_}Couldn't commit processed log positions"{_} (see [this discussion|https://debezium.zulipchat.com/#narrow/stream/302529-users/topic/Couldn't.20commit.20processed.20log.20positions.20with.20the.20source.20data])
h2. Do you see the same behaviour using the latest relesead Debezium version?

Didn't checked
h2. Do you have the connector logs, ideally from start till finish?

(You might be asked later to provide [DEBUG/TRACE|https://debezium.io/documentation/reference/stable/operations/logging.html] level log)

Exception itself
{code:json}
{
"exception": {
"stacktrace": "java.sql.SQLException: ORA-01284: file \/var\/opt\/.../MY_DB\/archive\/1_116358_1111587716.dbf cannot be opened
ORA-00308: cannot open archived log '\/var\/opt\/.../MY_DB\/archive\/1_116358_1111587716.dbf'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7
ORA-06512: at \"SYS.DBMS_LOGMNR\", line 68
ORA-06512: at line 1

at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:628)
...
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1034)
at oracle.jdbc.driver.OracleStatement.executeSQLStatement(OracleStatement.java:1507)
...
at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1010)
at io.debezium.connector.oracle.logminer.LogMinerHelper.executeCallableStatement(LogMinerHelper.java:217)
at io.debezium.connector.oracle.logminer.LogMinerHelper.setLogFilesForMining(LogMinerHelper.java:87)
at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.initializeRedoLogsForMining(LogMinerStreamingChangeEventSource.java:335)
at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:155)
at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:60)
at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:174)
at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:141)
at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:109)
at java.base\/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base\/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base\/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base\/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base\/java.lang.Thread.run(Thread.java:829)
Caused by: Error : 1284, Position : 0, Sql = BEGIN sys.dbms_logmnr.add_logfile(LOGFILENAME => '\/var\/opt/.../MY_DB\/archive\/1_116358_1111587716.dbf', OPTIONS => DBMS_LOGMNR.ADDFILE);END;, OriginalSql = BEGIN sys.dbms_logmnr.add_logfile(LOGFILENAME => '\/var\/opt\.../MY_DB\/archive\/1_116358_1111587716.dbf', OPTIONS => DBMS_LOGMNR.ADDFILE);END;, Error Msg = ORA-01284: file \/var\/opt/.../MY_DB\/archive\/1_116358_1111587716.dbf cannot be opened
ORA-00308: cannot open archived log '\/var\/opt/.../MY_DB\/archive\/1_116358_1111587716.dbf'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7
ORA-06512: at \"SYS.DBMS_LOGMNR\", line 68
ORA-06512: at line 1

at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:632)
... 26 more",
"exception_class": "java.sql.SQLException",
"exception_message": "ORA-01284: file \/var\/opt/.../MY_DB\/archive\/1_116358_1111587716.dbf cannot be opened
ORA-00308: cannot open archived log '\/var\/opt/.../MY_DB\/archive\/1_116358_1111587716.dbf'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7
ORA-06512: at \"SYS.DBMS_LOGMNR\", line 68
ORA-06512: at line 1
"
},
"source_host": "cmd-debezium-kafka-connect-6f56b6cdb9-dbt5c",
"method": "logError",
"level": "ERROR",
"message": "Mining session stopped due to the {}",
"mdc": {
"dbz.connectorContext": "streaming",
"dbz.connectorType": "Oracle",
"connector.context": "[oracle-customers-connector|task-0] ",
"dbz.taskId": "0",
"dbz.connectorName": "my-db"
},
"@timestamp": "2023-03-14T19:42:12.790Z",
"file": "LogMinerHelper.java",
"line_number": "117",
"thread_name": "debezium-oracleconnector-my-db-change-event-source-coordinator",
"@version": 1,
"logger_name": "io.debezium.connector.oracle.logminer.LogMinerHelper",
"class": "io.debezium.connector.oracle.logminer.LogMinerHelper"
}{code}
Causes repeatable restarts

!image-2023-03-17-19-58-58-784.png|width=600!

Full logs and thread dumps may be provided if the issue is not clear from the above snippets
h2. How to reproduce the issue using our [tutorial|https://github.com/debezium/debezium-examples/tree/main/tutorial] deployment?

Remove the archive log while the connector is running
h1. Feature request or enhancement

For feature requests or enhancements, provide this information, please:
h2. Which use case/requirement will be addressed by the proposed feature?
* Timely and easy detection of the issue - now it could be detected only by alerting on the absence of messages in Kafka topic or errors in logs. 
* Convenient troubleshooting - no misleading "{_}Couldn't commit processed log positions{_}" caused by livelock.

h2. Implementation ideas (optional)

Do not use retriable exception for this case in ErrorHandler

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.