airbytehq / airbytehq/airbyte

[DB sources] : Revisit checkpoint frequency for CDC sources

Aperta
#36,149 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
team/triage
Lingua principale
Python
Stelle
22.1k
Fork
5.3k
Metriche di merge delle PR
Metriche PR in attesa

Descrizione

### Topic

_No response_

### Relevant information

Many of Airbyte's data loss scenarios are based on an incorrect state message being sent. In CDC, and in Postgres this is especially troubling as the corresponding WAL is deleted.

Opening a ticket to :
1. Assess the effectiveness of state messages in CDC.
2. We think some of the logic can be simplified (remove time based checkpointing, reduce state emission frequency) because CDC syncs are recommended to happen frequently.
3. There are a lot of issues/edge cases with dealing with coordinating the state file with the event stream, so any simplification (or even removal) of state logic would help reduce data loss

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.