airbytehq / airbytehq/airbyte

Source MSSQL: CDC commit logs are not fetched properly

Ouverte
#25,237 7 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
area/connectors community connectors/source/mssql connectors/sources-database frozen releaseStage/beta team/db-dw-sources type/bug
Langage dominant
Python
Étoiles
22.1k
Forks
5.3k
Métriques de merge des PR
Métriques de PR en attente

Description

## Environment
- **Airbyte version**: 0.43.1
- **OS Version / Instance**: Container-Optimized OS with containerd (cos_containerd) / GKE
- **Deployment**: Google Kubernetes Engine
- **Source Connector and version**: airbyte/1.0.11
- **Destination Connector and version**: airbyte/destination-bigquery:1.2.20
- **Step where error happened**: Sync job

## Current Behavior
I integrate data from SQL Server 2017 with CDC enable to BigQuery, and configure "Incremental Deduped+History" on Airbyte.
Some of data are not integrated correctly.
These data would be modified multiple times in SQL Server in almost the same time, and the wrong order of commit would be kept in target destination.
As shown in following BigQuery photo, I rerun the SQLs generated when Airbyte syncing, found that the order would lead to wrong row of data kept.

## Expected Behavior
Keep correct data in target destination.
As Bigquery screenshot shown below, the correct data is the row with TC027 = 'Y', but the one with TC027 = 'U' is kept now.
Further, maybe Airbyte does not use column __$command_id in MSSQL CDC log screenshot to order the latest commit, so that the older commit is update to target destination.

## Logs
- MSSQL CDC log screenshot
![cdc logs](https://uuboyscy.notion.site/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2Ffd9454a6-5186-4821-8560-9e32e73ed389%2FUntitled.png?id=1830baf2-8f60-49fb-a065-520431cf036a&table=block&spaceId=6355b5e6-0aa0-4d0d-b335-a02c7d931bca&width=2000&userId=&cache=v2)

- Bigquery screenshot
![bigquery transformation temp table](https://uuboyscy.notion.site/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F4bae015b-4381-42e4-b544-eac28d53f286%2FUntitled.png?id=b90896ac-2d4e-4247-b6c4-d6e6a2039929&table=block&spaceId=6355b5e6-0aa0-4d0d-b335-a02c7d931bca&width=2000&userId=&cache=v2)

## Steps to Reproduce
As shown in above Bigquery screenshot, I rerun the SQLs generated when Airbyte syncing, found that the order would lead to wrong row of data kept (The data with TC027 = 'U' is kept, but TC027 = 'Y' is correct row).

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.