Aibyte replicating duplicte rows with Incrmental | Append mode
- Linguagem predominante
- Python
- Estrelas
- 22.1k
- Forks
- 5.4k
- Merge médio
- 5h
- PRs com merge (30d)
- 671
Descrição
### Topic
_No response_
### Relevant information
I tried to use Incremental Dedup | Append mode at the beginning for stores table to replicate from Postgres to Snowflake but syncs failed with an error related to NULL values in the _id column. The source table contains rows where _id is NULL, and Airbyte's dedup/normalization step could not process these — causing the sync to abort. I checked Postgres stores table and there was no _id column with null values. Airbyte source (postgres) setting was using update method as 'scan changes with user defined cursor'.
2. Switched to Incremental | Append — now seeing duplicate rows:
As a workaround we switched to Incremental|Append mode. Syncs now complete successfully, BUT we are seeing duplicate rows in Snowflake destination.
What's unusual: all duplicate rows for a given store have the exact same dateupdated value. The only difference between duplicate rows is _AIRBYTE_EXTRACTED_AT — spaced ~15 minutes apart (our sync frequency).
Example — 7 rows for same store in Snowflake, all identical data:
Idateupdated = 2026-07-14 12:33:17 _airbyte_extracted_at = 12:46 ← first sync
dateupdated = 2026-07-14 12:33:17 _airbyte_extracted_at = 13:03 ← duplicate
dateupdated = 2026-07-14 12:33:17 _airbyte_extracted_at = 13:17 ← duplicate
dateupdated = 2026-07-14 12:33:17 _airbyte_extracted_at = 13:33 ← duplicate
... and so on
This only affects a small number of stores (~16 out of 243K) at any given time. The affected stores change over time.
Question for community: What is causing Incremental|Append + UDC to replicate the same rows repeatedly with identical dateupdated? Is this a known behavior and is there a recommended fix short of switching to CDC?
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13108
Guia de contribuição
Direção de pesquisa
No source file or test is named. Start by reproducing the Postgres source configured with “scan changes with user defined cursor” and Incremental | Append, then inspect the sync logs and internal tracking issue 13108. Done means identifying why rows repeat and documenting a supported fix or confirmed behavior.
Escrita pelo modelo de indexação a partir do texto da issue.
Avaliação
- Stack de tecnologia
- postgresql
- Domínio
- data-engineering, databases
- Tipo de issue
- Bug
- Dificuldade
- 4/5
- Tempo estimado
- 3-5 dias
- Status de atividade
- Pouca atividade
- Clareza
- Precisa de esclarecimento
- Facilidade para iniciantes
- 35/100