Recover from older scn values with in-progress transactions more efficiently [DBZ-6097]
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from [DBZ-6097](https://issues.redhat.com/browse/DBZ-6097)
When the Oracle connector is restarted, the streaming phase begins to stream changes from the last known offset stored as {{scn}}. Suppose there were any in-progress transactions at the time the connector was restarted. In that case, this {{scn}} value could be from any point in time before the {{commit_scn}} stored in the offsets, and this requires that the connector re-read all changes from the logs since {{scn}} to build-up this "in-progress" buffer while also excluding all transactions that were committed or rolled back before the {{commit_scn}}.
This re-reading phase can be pretty expensive if the {{scn}} refers to a change well into the past as lots of logs and redo entries must be examined. Every redo entry is sent over the network back to Debezium to hydrate this in-memory buffer of the in-progress transaction state.
One possible way to extract this in-progress transaction state more efficiently would be to run an entirely separate query against LogMiner than what is used for the mining loop. This query would run against all logs since {{scn}} in a single LogMiner session and would automatically exclude any transactions that had been committed or rolled back. In effect, all Debezium would receive from the query's result set would be the redo entries that participate in the current set of in-progress transactions, up to the {{commit_scn}}.
Once the buffer is populated with the in-progress state, the connector will resume reading from {{commit_scn}}. Care will be necessary with Oracle RAC since each node's commit point is recorded separately.
Some open questions on this theory are:
# Would it be sufficient to use the most recent {{commit_scn}}?
# What if hundreds of logs are added in that in-progress query session? Do we need to split that up somehow? And if we split that up, how do we effectively handle aggregating the results across multiple queries and session batches?
Contributor guide
Assessment
This issue has not been assessed yet.