airbytehq / airbytehq/airbyte

[source-postgres] Incremental sync skips records when cursor field timestamp is extremely close to airbyte run start time

未關閉
#80,152 3 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
area/connectors autoteam community connectors/source/postgres needs-triage team/extensibility type/bug
主要語言
Python
星號
22.1k
分支
5.4k
平均合併
5 小時
30 天內合併 PR
671

描述

### Connector Name

source-postgres

### Connector Version

3.7.0

### What step the error happened?

During the sync

### Relevant information

**Summary**

During an incremental sync from Postgres, the connector missed two records that were inserted into the source database at almost the exact time the Airbyte sync started. The sync completed successfully with no errors, but those two records did not appear in the destination until we ran a full refresh.

**Environment**

- Airbyte deployment: OSS
- Source: Postgres
- Connector: source-postgres v3.7.0
- Destination: S3 Data Lake v0.3.44

**Sync configuration**

- Sync mode: Incremental
- Destination sync mode: Incremental - Append + Deduped
- Replication method:
- Standard (cursor based)
- Cursor field: `created_at`

**Whats happening + details**

Two records were created in the source table at the following cursor field (created_at) identical timestamps:

1. 2026-06-12 16:31:32.869
2. 2026-06-12 16:31:32.869

Here are the cursor values from the airbyte logs:
cursor value: 2026-06-12T16:01:35.607000 ---------  2026-06-12 12:31:33 run
cursor value: 2026-06-12T16:31:33.680000 ---------  2026-06-12 13:01:37 run

**Expected behavior**

All rows that exist in the source at or before the end of the incremental sync window should be replicated to the destination. Rows inserted around the sync time should not be permanently missed; they should either appear in that sync or in a subsequent incremental sync, without requiring a manual full refresh.

**Actual behavior**

- The incremental sync completed successfully with no errors.
- Two rows inserted at (or very close to) the sync start time were not replicated.
- Subsequent incremental syncs did not pick them up.
- Only a manual full refresh caused those rows to appear in the destination.

**Evidence / verification**

- Verified in Postgres that the two rows existed before and after the incremental sync.
- Verified in the destination that:
- They were missing after the incremental sync.
- They appeared only after the full refresh.
- No obvious errors or warnings in the Airbyte logs for the incremental sync.

**Logs**

[clinicaldataservice_logs_64692_txt.txt](https://github.com/user-attachments/files/29022914/clinicaldataservice_logs_64692_txt.txt)

### Relevant log output

```shell

```

### Contribute

- [ ] Yes, I want to contribute

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/12889

貢獻指南

開啟貢獻指南

研究方向

Start with the source-postgres connector's incremental, cursor-based replication path and review the attached logs around the two cursor timestamps and Airbyte run start times. Compare how records at the boundary are handled across subsequent incremental runs; done means rows present at or before the sync window end are replicated without requiring a full refresh.

由索引模型根據 Issue 內容生成。

評估

技術堆疊
postgresql
領域
data-engineering, databases
Issue 類型
缺陷
難度
4/5
預估耗時
3-5 天
活躍度
冷清
描述清晰度
基本清楚
新手友好度
48/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。