debezium / debezium/dbz

Improve handling of large batch loads with small SCN diffs [DBZ-9627]

Open
#1,134 0 comments 0 reactions 0 assignees View on GitHub
component/oracle-connector migrated-from-jira type/enhancement
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-9627](https://issues.redhat.com/browse/DBZ-9627)

Relates to this discussion here: [zulip|https://debezium.zulipchat.com/#narrow/channel/348250-community-oracle/topic/ORA-00310.20in.203.2E2.2E0.20during.20successive.20log.20switches]

h1. Report
Our DBA recently ran the below query to create an historical extract. They insert all the relevant rows to a table and then they create a flat file from the resulting table

{code:sql}
CREATE TABLE mms.branch_sku_day_27102025
TABLESPACE INDEX_MMS
NOLOGGING
AS
SELECT * FROM mms.branch_sku_day
WHERE last_update_date >= TO_DATE('2025-08-18', 'YYYY-MM-DD');
{code}

This is a very large table with around 28million rows per partition (partitioned per day). Once they realised how much data was being processed by this they killed the process from running. We're uncertain how much data this generated but it only ran for about 6 minutes.

On the DBZ we ended up crashing due to ORA-00310 which is a retriable error (this happens to us sometimes when there are many switches in quick succession). After the restart, queries start taking very long to run.

!image-2025-10-27-18-50-26-117.png|thumbnail!

h2. What Debezium connector do you use and what version?
3.2.0.Final with Debezium Oracle Connector

h2. What is the connector configuration?
These are our batch settings:
debezium.source.log.mining.batch.size.default=100000
debezium.source.log.mining.batch.size.min=10000
debezium.source.log.mining.batch.size.max=750000
debezium.source.log.mining.sleep.time.default.ms=200
debezium.source.log.mining.sleep.time.min.ms=0
debezium.source.log.mining.sleep.time.max.ms=1000
debezium.source.log.mining.sleep.time.increment.ms=100
debezium.source.log.mining.strategy=online_catalog
debezium.source.log.mining.query.filter.mode=in
debezium.source.log.mining.transaction.retention.ms=43200000
debezium.source.query.fetch.size=100000

h2. What is the captured database version and mode of deployment?
Oracle 19c with Debezium Server. Logs are each 9gb

The hope is that we can compute the data mining range in a different way to better handle this burst of redo logs. Once we get past this transaction in the logs DBZ's behaviour seems more normal even though we remain many logs behind.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.