airbytehq / airbytehq/airbyte

[DB sources] : Revisit checkpoint frequency for CDC sources

未關閉
#36,149 1 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
team/triage
主要語言
Python
星號
22.1k
分支
5.4k
平均合併
5 小時
30 天內合併 PR
671

描述

### Topic

_No response_

### Relevant information

Many of Airbyte's data loss scenarios are based on an incorrect state message being sent. In CDC, and in Postgres this is especially troubling as the corresponding WAL is deleted.

Opening a ticket to :
1. Assess the effectiveness of state messages in CDC.
2. We think some of the logic can be simplified (remove time based checkpointing, reduce state emission frequency) because CDC syncs are recommended to happen frequently.
3. There are a lot of issues/edge cases with dealing with coordinating the state file with the event stream, so any simplification (or even removal) of state logic would help reduce data loss

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。