debezium / debezium/dbz

Allow pinning the mining session window using only promoted transactions in deferred mode

Open
#2,166 0 comments 0 reactions 1 assignee Claimed by @nathan-smit-1 View on GitHub
component/oracle-connector type/enhancement
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

## Feature request or enhancement

In 3.6 we have added a configuration that allows for "deferred" transaction start. The transaction metadata is stored in a map and only promoted to a true transaction once DML is observed. As part of that change, the mining session window is never pinned as we were looking to emulate the behavior of legacy.transaction.start from earlier DBZ versions.

After this, Chris suggested we investigate an enhancement to this where we emulate the default "pinning" behaviour where the mining session gets pinned but only by transactions that are promoted. This is a potential middle-ground where users can avoid edge-cases caused by not mining the entire transaction in one pass but in a narrow way that only involves transactions that are "important." I have been doing some work on this and I'm busy doing testing at the moment.

---

**Implementation ideas (optional)**

Core of the idea is: Pin the mining session using the eldest transaction in the transaction cache. When a transaction is promoted from the deferred cache, it will then pin the window.

Caveat, in my current work on this, I have done the following: When a transaction is promoted and its START_SCN is older than eldest_scn in the transaction cache, the mining session window widens. However, I am also currently expanding the fetch query window and re-mining that segment in its entirety. The concern being that in the pass where you retrieve the DML there is some edge-case (missing commit, unsupported operation etc) and we need to remine that segment using the full session window in order to mine correctly. The reason work to handle recent duplicates should help with this but I'm still thinking through implications

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.