debezium / debezium/dbz

Recover from older scn values with in-progress transactions more efficiently [DBZ-6097]

Open
#760 0 comments 0 reactions 1 assignee Claimed by @Naros View on GitHub
component/oracle-connector migrated-from-jira type/task
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-6097](https://issues.redhat.com/browse/DBZ-6097)

When the Oracle connector is restarted, the streaming phase begins to stream changes from the last known offset stored as {{scn}}. Suppose there were any in-progress transactions at the time the connector was restarted. In that case, this {{scn}} value could be from any point in time before the {{commit_scn}} stored in the offsets, and this requires that the connector re-read all changes from the logs since {{scn}} to build-up this "in-progress" buffer while also excluding all transactions that were committed or rolled back before the {{commit_scn}}.

This re-reading phase can be pretty expensive if the {{scn}} refers to a change well into the past as lots of logs and redo entries must be examined. Every redo entry is sent over the network back to Debezium to hydrate this in-memory buffer of the in-progress transaction state.

One possible way to extract this in-progress transaction state more efficiently would be to run an entirely separate query against LogMiner than what is used for the mining loop. This query would run against all logs since {{scn}} in a single LogMiner session and would automatically exclude any transactions that had been committed or rolled back. In effect, all Debezium would receive from the query's result set would be the redo entries that participate in the current set of in-progress transactions, up to the {{commit_scn}}.

Once the buffer is populated with the in-progress state, the connector will resume reading from {{commit_scn}}. Care will be necessary with Oracle RAC since each node's commit point is recorded separately.

Some open questions on this theory are:
# Would it be sufficient to use the most recent {{commit_scn}}?
# What if hundreds of logs are added in that in-progress query session? Do we need to split that up somehow? And if we split that up, how do we effectively handle aggregating the results across multiple queries and session batches?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.