debezium / debezium/dbz

Hidden columns cause schema mismatch errors during an incremental snapshot [DBZ-8430]

Open
#1,317 0 comments 0 reactions 0 assignees View on GitHub
component/core-library migrated-from-jira type/bug
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-8430](https://issues.redhat.com/browse/DBZ-8430)

h2. What Debezium connector do you use and what version?

SQL Server, 2.7.3.Final
h2. Do you see the same behaviour using the latest released Debezium version?

Yes
h2. What behavior do you see?

When a table includes columns which are marked as *Hidden* e.g.
{code:java}
CREATE TABLE niagaradb.niagaratest.HiddenTest
(
DeptID INT NOT NULL PRIMARY KEY CLUSTERED,
DeptName VARCHAR(50) NOT NULL,
ManagerID INT NULL,
ParentDeptID INT NULL,
SysStartTime DATETIME2 GENERATED ALWAYS AS ROW START HIDDEN NOT NULL,
SysEndTime DATETIME2 GENERATED ALWAYS AS ROW END HIDDEN NOT NULL,
PERIOD FOR SYSTEM_TIME (SysStartTime, SysEndTime)
)
WITH (SYSTEM_VERSIONING = ON); {code}
If you trigger an incremental snapshot on the table, you will encounter a schema mismatch error:

 
{code:java}
2024-11-15T09:56:24.779Z  WARN 1 --- [rce-coordinator] i.d.pipeline.signal.SignalProcessor      : Action snapshot-window-close failed. The signal SignalRecord{id='8f354677-7179-499b-a68c-39b95804e480-close', type='snapshot-window-close', data='{"openWindowTimestamp": "2024-11-15T09:56:19.815216984Z", "closeWindowTimestamp": "2024-11-15T09:56:19.843987916Z"}', additionalData={}} may not have been processed.
org.apache.kafka.connect.errors.ConnectException: Data row is smaller than a column index, internal schema representation is probably out of sync with real database schema
{code}
 

This occurs due to the fact that (by default) an incremental snapshot uses *Select ** statements throughout the process, meaning the data for the hidden columns is not returned by the query. 

The hidden columns work as intended during an initial snapshot as we use a *Select * statement. 

The hidden columns work as intended when streaming change events as they are included in the sql server cdc tables.
h2. Implementation ideas (optional)

I believe we should rewrite *AbstractChunkQueryBuilder#buildProjection* to always use a list of columns in the incremental snapshot select statements. e.g.
{code:java}
protected String buildProjection(Table table) {
TableId tableId = table.id();
return table.columns().stream()
.filter(column -> !connectorConfig.isColumnsFiltered() || columnFilter.matches(tableId.catalog(), tableId.schema(), tableId.table(), column.name()))
.map(column -> jdbcConnection.quotedColumnIdString(column.name()))
.collect(Collectors.joining(", "));
} {code}

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.