Source MSSQL: CDC commit logs are not fetched properly
- Langage dominant
- Python
- Étoiles
- 22.1k
- Forks
- 5.3k
- Métriques de merge des PR
- Métriques de PR en attente
Description
## Environment
- **Airbyte version**: 0.43.1
- **OS Version / Instance**: Container-Optimized OS with containerd (cos_containerd) / GKE
- **Deployment**: Google Kubernetes Engine
- **Source Connector and version**: airbyte/1.0.11
- **Destination Connector and version**: airbyte/destination-bigquery:1.2.20
- **Step where error happened**: Sync job
## Current Behavior
I integrate data from SQL Server 2017 with CDC enable to BigQuery, and configure "Incremental Deduped+History" on Airbyte.
Some of data are not integrated correctly.
These data would be modified multiple times in SQL Server in almost the same time, and the wrong order of commit would be kept in target destination.
As shown in following BigQuery photo, I rerun the SQLs generated when Airbyte syncing, found that the order would lead to wrong row of data kept.
## Expected Behavior
Keep correct data in target destination.
As Bigquery screenshot shown below, the correct data is the row with TC027 = 'Y', but the one with TC027 = 'U' is kept now.
Further, maybe Airbyte does not use column __$command_id in MSSQL CDC log screenshot to order the latest commit, so that the older commit is update to target destination.
## Logs
- MSSQL CDC log screenshot

- Bigquery screenshot

## Steps to Reproduce
As shown in above Bigquery screenshot, I rerun the SQLs generated when Airbyte syncing, found that the order would lead to wrong row of data kept (The data with TC027 = 'U' is kept, but TC027 = 'Y' is correct row).
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.