Aibyte replicating duplicte rows with Incrmental | Append mode
- 主要言語
- Python
- スター
- 22.1k
- フォーク
- 5.3k
- PR マージ指標
- PR 指標を取得中
説明
### Topic
_No response_
### Relevant information
I tried to use Incremental Dedup | Append mode at the beginning for stores table to replicate from Postgres to Snowflake but syncs failed with an error related to NULL values in the _id column. The source table contains rows where _id is NULL, and Airbyte's dedup/normalization step could not process these — causing the sync to abort. I checked Postgres stores table and there was no _id column with null values. Airbyte source (postgres) setting was using update method as 'scan changes with user defined cursor'.
2. Switched to Incremental | Append — now seeing duplicate rows:
As a workaround we switched to Incremental|Append mode. Syncs now complete successfully, BUT we are seeing duplicate rows in Snowflake destination.
What's unusual: all duplicate rows for a given store have the exact same dateupdated value. The only difference between duplicate rows is _AIRBYTE_EXTRACTED_AT — spaced ~15 minutes apart (our sync frequency).
Example — 7 rows for same store in Snowflake, all identical data:
Idateupdated = 2026-07-14 12:33:17 _airbyte_extracted_at = 12:46 ← first sync
dateupdated = 2026-07-14 12:33:17 _airbyte_extracted_at = 13:03 ← duplicate
dateupdated = 2026-07-14 12:33:17 _airbyte_extracted_at = 13:17 ← duplicate
dateupdated = 2026-07-14 12:33:17 _airbyte_extracted_at = 13:33 ← duplicate
... and so on
This only affects a small number of stores (~16 out of 243K) at any given time. The affected stores change over time.
Question for community: What is causing Incremental|Append + UDC to replicate the same rows repeatedly with identical dateupdated? Is this a known behavior and is there a recommended fix short of switching to CDC?
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13108
コントリビューションガイド
評価
この issue はまだ評価されていません。