redpanda-data / redpanda-data/connect

microsoft_sql_server_cdc: use cluster index when primary key is missing

Open
#4,324 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
8.8k
Forks
969
Avg merge
1d 13h
Merged PRs (30d)
64

Description

Hi,

When using snapshot we can use cluster index instead a primary key.
We can try additional query here to search cluster index.

As alternative, to add config field snapshot_order_columns that defines order by columns.

To find cluster index:

SELECT t.name,  c.name FROM sys.indexes AS i
  JOIN sys.index_columns ic ON i.object_id = ic.object_id AND i.index_id = ic.index_id
  JOIN sys.columns c ON ic.object_id = c.object_id AND ic.column_id = c.column_id
  JOIN sys.tables t ON i.object_id = t.object_id
  JOIN sys.schemas s ON t.schema_id = s.schema_id
WHERE i.type = 1 -- 1 is cluster
 AND s.name = ? -- schema
 AND t.name = ? -- table
ORDER BY ic.key_ordinal;

Regards

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in internal/impl/mssqlserver/replication/snapshot.go at the linked lookup around line 251, and trace how snapshot ordering is chosen when a primary key is absent. Use the proposed sys.indexes query as the reference for finding clustered-index columns. Done means snapshots can use the clustered index when no primary key exists.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
data-engineering, databases, stream-processing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.