Source Snowflake: Cursor Row Count Discrepencies Reuploads Previous Package
- Dominant language
- Python
- Stars
- 22.1k
- Forks
- 5.3k
- PR merge metrics
- PR metrics pending
Description
### Connector Name
source-snowflake
### Connector Version
0.2.0
### What step the error happened?
During the sync
### Relevant information
Initial Message Here: https://airbytehq.slack.com/archives/C021JANJ6TY/p1696018540579129
For Incremental|Append syncs, the Snowflake source does a cursor count check between the connector's metadata and the current Snowflake source. If it finds any difference, it re-uploads the entire previous package (all data with that previous cursor). This makes sense when data is updated/modified or added. However, it doesn't make sense when data is removed.
In our current work, we use the Snowflake source and a custom destination to sync data on subjects in a study. Because of the work we do, some subjects become excluded from the study and we are able to (and would like to) manually remove them from the source & destination without the connector seeing this row count discrepancy and re-loading the entire package. Every time this occurs, we need to delete all data from the destination and re-upload all of it, making this work extremely difficult to maintain.
Goal:
- Row count decreases don't cause a re-upload of records
- Or, an optional flag is added so we can turn this off
We may be willing to contribute if necessary
### Relevant log output
```shell
Found matching cursor in state. Stream: _. Cursor Field: PACKAGE_ID Value: 20230424 Count: 1459
Table cursor count: expected 1459, actual 1458
```
### Contribute
- [ ] Yes, I want to contribute
Contributor guide
Assessment
This issue has not been assessed yet.