airbytehq / airbytehq/airbyte

Gorgias source - incremental sync re-syncing duplicates, runs for hours

未關閉
#80,326 3 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
area/connectors autoteam community needs-triage team/extensibility team/use type/bug
主要語言
Python
星號
22.1k
分支
5.4k
平均合併
5 小時
30 天內合併 PR
671

描述

### Connector Name

Gorgias

### Connector Version

0.1.42

### What step the error happened?

During the sync

### Relevant information

We're experiencing an issue with our Gorgias source connector where incremental sync is not behaving incrementally. Each sync re-pulls records that were already synced in prior runs, producing large volumes of duplicate data in the destination and causing syncs to run for several hours instead of completing quickly.

Connection details:
- Source: Gorgias
- Destination: BigQuery
- Affected stream(s): [e.g. messages, tickets]
- Sync mode: [Incremental | Append OR Incremental | Append + Deduped]
- Cursor field: [ created_datetime]
- Environment: [Airbyte Cloud]
- Gorgias source connector version: 0.1.42

What we expect:
After an initial full sync, subsequent incremental runs should only fetch records created or updated since the last cursor value, and complete in minutes.

What's actually happening:
- Every sync re-fetches a large overlapping set of records that were already present from previous runs (duplicates).
- Sync durations have grown to several hours.
- The cursor / state does not appear to be advancing or filtering correctly between runs.

We're currently deduplicating downstream in BigQuery as a workaround, but we'd like the incremental sync itself to behave correctly to reduce sync time and cost.

### Relevant log output

```shell

```

### Contribute

- [ ] Yes, I want to contribute

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/12944

貢獻指南

開啟貢獻指南

研究方向

The report identifies the Gorgias source connector version 0.1.42, incremental sync state and cursor handling, and affected streams such as messages and tickets, but provides no logs or file paths. Start by reproducing an incremental run after a full sync and inspect whether the created_datetime cursor advances and filters records. Done means later runs fetch only records created or updated after the saved cursor without duplicates.

由索引模型根據 Issue 內容生成。

評估

技術堆疊
python
領域
data-engineering
Issue 類型
缺陷
難度
4/5
預估耗時
3-5 天
活躍度
冷清
描述清晰度
基本清楚
新手友好度
38/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。