airbytehq / airbytehq/airbyte

[source-instagram] Add comments stream & some improvements

Aperta
#61,481 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
area/connectors autoteam community connectors/source/instagram team/extensibility type/enhancement
Lingua principale
Python
Stelle
22.1k
Fork
5.3k
Metriche di merge delle PR
Metriche PR in attesa

Descrizione

### Topic

Add comments stream & some improvements

### Relevant information

In my fork we added the comments stream, similar to the media_insights. For each media, get the comments. Just opening the issue to know if I can make a PR with it.

But also, now that I have been working closely with the instagram connector I noticed some things I find weird.

For example, today it will always download **all** media and stories for the accounts, and then **all** insights for them (of course there's incremental for insights but either way, still for all media and stories). There is a `start_date´ config in the stream, but its only for user insights. And when no start date is provided, it downloads user insights for the past 2 years (this is hardcoded). This made some syncs be unnecessarily long, because we only wanted data for the past year or so.

Our view on this is that `start_date` should apply to media and stories stream as well. I made that change in our fork, but to keep the current behaviour added separate configs to download all media and all stories, in case that's what the user wants. To keep compatibility maybe those should be set to true by default.

And also, as improvement, added the option for the user to configure the source to download data from a fixed period, not a fixed start date. So we can set it to always download media & etc from the past 30 days, for example. This also removes the hardcoded 2 years default.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.