airbytehq / airbytehq/airbyte

[source-outlook] conversations stream lacks incremental sync and Full Refresh | Overwrite + Deduped creates massive duplicate records

未關閉
#83,476 3 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
area/connectors autoteam community connectors/source/outlook needs-triage team/extensibility team/use type/bug
主要語言
Python
星號
22.1k
分支
5.4k
平均合併
5 小時
30 天內合併 PR
671

描述

### Connector Name

source-outlook

### Connector Version

0.0.26

### What step the error happened?

During the sync

### Relevant information

The Microsoft Outlook source connector is creating massive duplication when syncing the conversations stream.

Although my Outlook account only contains 10–15 unique conversations, the destination table grows to 300,000–400,000 records after repeated syncs.

**Current Behavior:**
Initial sync completes successfully and imports approximately 10–15 conversations.
Every subsequent sync re-fetches the same conversations.
Even though conversationId is configured as the primary key and Overwrite + Deduped is selected, duplicate records continue accumulating.
The destination eventually contains 300,000–400,000 rows despite only 10–15 unique conversation IDs.
I reproduced the same behavior across multiple Airbyte accounts/workspaces, suggesting this is a connector-level issue rather than a workspace-specific configuration issue.
Expected Behavior

**Suspected Root Cause:**
The conversations stream emits one record per message because **$groupby: conversationId** in [its yaml manifest](https://github.com/airbytehq/airbyte/blob/a6e2c7ce/airbyte-integrations/connectors/source-outlook/manifest.yaml#L456-L470) is not a valid Microsoft Graph OData parameter on /me/messages and is silently ignored, so it never actually returns unique conversations.

### Relevant log output

```shell
3 Aug 3 Aug

Active Streams 116,116 records loaded Search

Status Stream name Latest sync in records Data fresh as of
(Sum over attempts)

Syncing conversations 116,011 extracted | 19m elapsed a few seconds ago
Syncing mailboxes 8 extracted | 19m elapsed a few seconds ago
Syncing messages 48 extracted | 19m elapsed a few seconds ago
Syncing messages_details 48 extracted | 19m elapsed a few seconds ago
Syncing profile 1 extracted | 19m elapsed a few seconds ago
```

### Contribute

- [ ] Yes, I want to contribute

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13232

貢獻指南

開啟貢獻指南

研究方向

Inspect the conversations stream in manifest.yaml around lines 456-470, then review how Microsoft Graph /me/messages handles the cited grouping parameter and repeated syncs. Reproduce the sync using the log pattern in the issue and verify that subsequent syncs do not create duplicate conversation records, including the requested incremental sync and full-refresh behavior.

由索引模型根據 Issue 內容生成。

評估

技術堆疊
python
領域
data-engineering
Issue 類型
缺陷
難度
4/5
預估耗時
3-5 天
活躍度
冷清
描述清晰度
基本清楚
新手友好度
55/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。