[source-instagram] Add comments stream & some improvements
- Lingua principale
- Python
- Stelle
- 22.1k
- Fork
- 5.3k
- Metriche di merge delle PR
- Metriche PR in attesa
Descrizione
### Topic
Add comments stream & some improvements
### Relevant information
In my fork we added the comments stream, similar to the media_insights. For each media, get the comments. Just opening the issue to know if I can make a PR with it.
But also, now that I have been working closely with the instagram connector I noticed some things I find weird.
For example, today it will always download **all** media and stories for the accounts, and then **all** insights for them (of course there's incremental for insights but either way, still for all media and stories). There is a `start_date´ config in the stream, but its only for user insights. And when no start date is provided, it downloads user insights for the past 2 years (this is hardcoded). This made some syncs be unnecessarily long, because we only wanted data for the past year or so.
Our view on this is that `start_date` should apply to media and stories stream as well. I made that change in our fork, but to keep the current behaviour added separate configs to download all media and all stories, in case that's what the user wants. To keep compatibility maybe those should be set to true by default.
And also, as improvement, added the option for the user to configure the source to download data from a fixed period, not a fixed start date. So we can set it to always download media & etc from the past 30 days, for example. This also removes the hardcoded 2 years default.
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.