[DB sources] : Revisit checkpoint frequency for CDC sources
未關閉
team/triage
- 主要語言
- Python
- 星號
- 22.1k
- 分支
- 5.4k
- 平均合併
- 5 小時
- 30 天內合併 PR
- 671
描述
### Topic
_No response_
### Relevant information
Many of Airbyte's data loss scenarios are based on an incorrect state message being sent. In CDC, and in Postgres this is especially troubling as the corresponding WAL is deleted.
Opening a ticket to :
1. Assess the effectiveness of state messages in CDC.
2. We think some of the logic can be simplified (remove time based checkpointing, reduce state emission frequency) because CDC syncs are recommended to happen frequently.
3. There are a lot of issues/edge cases with dealing with coordinating the state file with the event stream, so any simplification (or even removal) of state logic would help reduce data loss
貢獻指南
評估
這個 Issue 還沒有評估資料。