airbytehq / airbytehq/airbyte

[source-postgres] Incremental sync skips records when cursor field timestamp is extremely close to airbyte run start time

Aberta
#80,152 3 comentários 0 reações 0 responsáveis Ver no GitHub
area/connectors autoteam community connectors/source/postgres needs-triage team/extensibility type/bug
Linguagem predominante
Python
Estrelas
22.1k
Forks
5.3k
Métricas de merge de PRs
Métricas de PR pendentes

Descrição

### Connector Name

source-postgres

### Connector Version

3.7.0

### What step the error happened?

During the sync

### Relevant information

**Summary**

During an incremental sync from Postgres, the connector missed two records that were inserted into the source database at almost the exact time the Airbyte sync started. The sync completed successfully with no errors, but those two records did not appear in the destination until we ran a full refresh.

**Environment**

- Airbyte deployment: OSS
- Source: Postgres
- Connector: source-postgres v3.7.0
- Destination: S3 Data Lake v0.3.44

**Sync configuration**

- Sync mode: Incremental
- Destination sync mode: Incremental - Append + Deduped
- Replication method:
- Standard (cursor based)
- Cursor field: `created_at`

**Whats happening + details**

Two records were created in the source table at the following cursor field (created_at) identical timestamps:

1. 2026-06-12 16:31:32.869
2. 2026-06-12 16:31:32.869

Here are the cursor values from the airbyte logs:
cursor value: 2026-06-12T16:01:35.607000 ---------  2026-06-12 12:31:33 run
cursor value: 2026-06-12T16:31:33.680000 ---------  2026-06-12 13:01:37 run

**Expected behavior**

All rows that exist in the source at or before the end of the incremental sync window should be replicated to the destination. Rows inserted around the sync time should not be permanently missed; they should either appear in that sync or in a subsequent incremental sync, without requiring a manual full refresh.

**Actual behavior**

- The incremental sync completed successfully with no errors.
- Two rows inserted at (or very close to) the sync start time were not replicated.
- Subsequent incremental syncs did not pick them up.
- Only a manual full refresh caused those rows to appear in the destination.

**Evidence / verification**

- Verified in Postgres that the two rows existed before and after the incremental sync.
- Verified in the destination that:
- They were missing after the incremental sync.
- They appeared only after the full refresh.
- No obvious errors or warnings in the Airbyte logs for the incremental sync.

**Logs**

[clinicaldataservice_logs_64692_txt.txt](https://github.com/user-attachments/files/29022914/clinicaldataservice_logs_64692_txt.txt)

### Relevant log output

```shell

```

### Contribute

- [ ] Yes, I want to contribute

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/12889

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.