[DB sources] : Revisit checkpoint frequency for CDC sources
- Lingua principale
- Python
- Stelle
- 22.1k
- Fork
- 5.3k
- Metriche di merge delle PR
- Metriche PR in attesa
Descrizione
### Topic
_No response_
### Relevant information
Many of Airbyte's data loss scenarios are based on an incorrect state message being sent. In CDC, and in Postgres this is especially troubling as the corresponding WAL is deleted.
Opening a ticket to :
1. Assess the effectiveness of state messages in CDC.
2. We think some of the logic can be simplified (remove time based checkpointing, reduce state emission frequency) because CDC syncs are recommended to happen frequently.
3. There are a lot of issues/edge cases with dealing with coordinating the state file with the event stream, so any simplification (or even removal) of state logic would help reduce data loss
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.