airbytehq / airbytehq/airbyte

Source Snowflake: Cursor Row Count Discrepencies Reuploads Previous Package

Open
#32,115 3 comments 0 reactions 0 assignees View on GitHub
area/connectors community connectors/source/snowflake frozen team/db-dw-sources type/bug
Dominant language
Python
Stars
22.1k
Forks
5.3k
PR merge metrics
PR metrics pending

Description

### Connector Name

source-snowflake

### Connector Version

0.2.0

### What step the error happened?

During the sync

### Relevant information

Initial Message Here: https://airbytehq.slack.com/archives/C021JANJ6TY/p1696018540579129

For Incremental|Append syncs, the Snowflake source does a cursor count check between the connector's metadata and the current Snowflake source. If it finds any difference, it re-uploads the entire previous package (all data with that previous cursor). This makes sense when data is updated/modified or added. However, it doesn't make sense when data is removed.

In our current work, we use the Snowflake source and a custom destination to sync data on subjects in a study. Because of the work we do, some subjects become excluded from the study and we are able to (and would like to) manually remove them from the source & destination without the connector seeing this row count discrepancy and re-loading the entire package. Every time this occurs, we need to delete all data from the destination and re-upload all of it, making this work extremely difficult to maintain.

Goal:
- Row count decreases don't cause a re-upload of records
- Or, an optional flag is added so we can turn this off

We may be willing to contribute if necessary

### Relevant log output

```shell
Found matching cursor in state. Stream: _. Cursor Field: PACKAGE_ID Value: 20230424 Count: 1459

Table cursor count: expected 1459, actual 1458
```

### Contribute

- [ ] Yes, I want to contribute

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.