[source-outlook] conversations stream lacks incremental sync and Full Refresh | Overwrite + Deduped creates massive duplicate records
- 主要語言
- Python
- 星號
- 22.1k
- 分支
- 5.4k
- 平均合併
- 5 小時
- 30 天內合併 PR
- 671
描述
### Connector Name
source-outlook
### Connector Version
0.0.26
### What step the error happened?
During the sync
### Relevant information
The Microsoft Outlook source connector is creating massive duplication when syncing the conversations stream.
Although my Outlook account only contains 10–15 unique conversations, the destination table grows to 300,000–400,000 records after repeated syncs.
**Current Behavior:**
Initial sync completes successfully and imports approximately 10–15 conversations.
Every subsequent sync re-fetches the same conversations.
Even though conversationId is configured as the primary key and Overwrite + Deduped is selected, duplicate records continue accumulating.
The destination eventually contains 300,000–400,000 rows despite only 10–15 unique conversation IDs.
I reproduced the same behavior across multiple Airbyte accounts/workspaces, suggesting this is a connector-level issue rather than a workspace-specific configuration issue.
Expected Behavior
**Suspected Root Cause:**
The conversations stream emits one record per message because **$groupby: conversationId** in [its yaml manifest](https://github.com/airbytehq/airbyte/blob/a6e2c7ce/airbyte-integrations/connectors/source-outlook/manifest.yaml#L456-L470) is not a valid Microsoft Graph OData parameter on /me/messages and is silently ignored, so it never actually returns unique conversations.
### Relevant log output
```shell
3 Aug 3 Aug
Active Streams 116,116 records loaded Search
Status Stream name Latest sync in records Data fresh as of
(Sum over attempts)
Syncing conversations 116,011 extracted | 19m elapsed a few seconds ago
Syncing mailboxes 8 extracted | 19m elapsed a few seconds ago
Syncing messages 48 extracted | 19m elapsed a few seconds ago
Syncing messages_details 48 extracted | 19m elapsed a few seconds ago
Syncing profile 1 extracted | 19m elapsed a few seconds ago
```
### Contribute
- [ ] Yes, I want to contribute
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13232
貢獻指南
研究方向
Inspect the conversations stream in manifest.yaml around lines 456-470, then review how Microsoft Graph /me/messages handles the cited grouping parameter and repeated syncs. Reproduce the sync using the log pattern in the issue and verify that subsequent syncs do not create duplicate conversation records, including the requested incremental sync and full-refresh behavior.
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- python
- 領域
- data-engineering
- Issue 類型
- 缺陷
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 活躍度
- 冷清
- 描述清晰度
- 基本清楚
- 新手友好度
- 55/100