airbytehq / airbytehq/airbyte

Aibyte replicating duplicte rows with Incrmental | Append mode

オープン
#82,131 コメント 5 件 リアクション 0 件 担当者 0 名 GitHub で見る
autoteam community needs-triage team/use
主要言語
Python
スター
22.1k
フォーク
5.3k
PR マージ指標
PR 指標を取得中

説明

### Topic

_No response_

### Relevant information

I tried to use Incremental Dedup | Append mode at the beginning for stores table to replicate from Postgres to Snowflake but syncs failed with an error related to NULL values in the _id column. The source table contains rows where _id is NULL, and Airbyte's dedup/normalization step could not process these — causing the sync to abort. I checked Postgres stores table and there was no _id column with null values. Airbyte source (postgres) setting was using update method as 'scan changes with user defined cursor'.

2. Switched to Incremental | Append — now seeing duplicate rows:
As a workaround we switched to Incremental|Append mode. Syncs now complete successfully, BUT we are seeing duplicate rows in Snowflake destination.
What's unusual: all duplicate rows for a given store have the exact same dateupdated value. The only difference between duplicate rows is _AIRBYTE_EXTRACTED_AT — spaced ~15 minutes apart (our sync frequency).
Example — 7 rows for same store in Snowflake, all identical data:
Idateupdated = 2026-07-14 12:33:17 _airbyte_extracted_at = 12:46 ← first sync
dateupdated = 2026-07-14 12:33:17 _airbyte_extracted_at = 13:03 ← duplicate
dateupdated = 2026-07-14 12:33:17 _airbyte_extracted_at = 13:17 ← duplicate
dateupdated = 2026-07-14 12:33:17 _airbyte_extracted_at = 13:33 ← duplicate
... and so on
This only affects a small number of stores (~16 out of 243K) at any given time. The affected stores change over time.
Question for community: What is causing Incremental|Append + UDC to replicate the same rows repeatedly with identical dateupdated? Is this a known behavior and is there a recommended fix short of switching to CDC?

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13108

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。