airbytehq / airbytehq/airbyte

Support incremental sync for Recurly

Aperta
#1,384 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
area/connectors autoteam difficulty - :star: :star: Icebox team/extensibility team/marketplace type/enhancement
Lingua principale
Python
Stelle
22.1k
Fork
5.3k
Metriche di merge delle PR
Metriche PR in attesa

Descrizione

## Tell us about the problem you're trying to solve
Follow up to airbytehq/airbyte-internal-issues#21

There are a few parts to this:

1. Identify which streams can sync incrementally, and which bookmark you'll be using
1. Update the catalog generated in `discover` to declare which streams sync incrementally
1. Implement the incremental sync process

To incrementally sync a child entity child entity (e.g: `/customers/{customer_id}/customer_events` ), we will probably need to do a "full read" of the parent entity (get all parents), then look for the child entities that have changed since the last time we read the child entities. For example: if we're syncing `/customers/{customer_id}/customer_events` then we should read all customers, then for each customer, read all the events we haven't read since the last time we performed a sync. It's probably not practical to keep track of the last event we synced for _each_ customer, so it may make sense to keep track of a global cursor for all customer_events. For example, we may just have to remember that the largest `customer_event.updated_at` value we extracted last time a sync ran was `october 10`. In which case the next sync we read all customers, then for each customer extract all the events where `updated_at >= october 10`.

┆Issue is synchronized with this [Asana task](https://app.asana.com/0/1200367912513076/1200368203057302) by [Unito](https://www.unito.io)

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.