airbytehq / airbytehq/airbyte

[source-instagram] Add comments stream & some improvements

未關閉
#61,481 1 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
area/connectors autoteam community connectors/source/instagram team/extensibility type/enhancement
主要語言
Python
星號
22.1k
分支
5.4k
平均合併
5 小時
30 天內合併 PR
671

描述

### Topic

Add comments stream & some improvements

### Relevant information

In my fork we added the comments stream, similar to the media_insights. For each media, get the comments. Just opening the issue to know if I can make a PR with it.

But also, now that I have been working closely with the instagram connector I noticed some things I find weird.

For example, today it will always download **all** media and stories for the accounts, and then **all** insights for them (of course there's incremental for insights but either way, still for all media and stories). There is a `start_date´ config in the stream, but its only for user insights. And when no start date is provided, it downloads user insights for the past 2 years (this is hardcoded). This made some syncs be unnecessarily long, because we only wanted data for the past year or so.

Our view on this is that `start_date` should apply to media and stories stream as well. I made that change in our fork, but to keep the current behaviour added separate configs to download all media and all stories, in case that's what the user wants. To keep compatibility maybe those should be set to true by default.

And also, as improvement, added the option for the user to configure the source to download data from a fixed period, not a fixed start date. So we can set it to always download media & etc from the past 30 days, for example. This also removes the hardcoded 2 years default.

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。