airbytehq / airbytehq/PyAirbyte

Support aliased Destinations or Sources (for State Caches)

未關閉
#570 3 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
accepting pull requests
主要語言
Python
星號
344
分支
77
平均合併
1 天 11 小時
30 天內合併 PR
35

描述

The current implementation of the [DestinationStreamStateModel](https://github.com/airbytehq/PyAirbyte/blob/f7b88eba400d7aa768c8c370cfbac6f18dfc61c6/airbyte/caches/_state_backend.py#L68) assumes a one-to-one relationship between a [Source](https://github.com/airbytehq/PyAirbyte/blob/f7b88eba400d7aa768c8c370cfbac6f18dfc61c6/airbyte/sources/base.py#L52) name and a [Destination](https://github.com/airbytehq/PyAirbyte/blob/f7b88eba400d7aa768c8c370cfbac6f18dfc61c6/airbyte/destinations/base.py#L38) name. This is fine in principle, under the assumption that the Connection only has one instance of the Source type. However, this is not always the case, lets imagine we have multiple cases of a Google Sheets Source, for which we want to keep its state tracked via `state_cache` when writing to a destination. How can we achieve this?

In Airbyte, we achieve this by aliasing the Sources and Destinations:
![Image](https://github.com/user-attachments/assets/284ad34a-ccc3-4160-9ace-6466cf605e39)
![Image](https://github.com/user-attachments/assets/39809b3b-25a1-4013-8b81-712fda922947)

By separating the ORM model to write based on the aliased Source and Destination/Cache names, we can support multiple types of Sources and Destinations and still hold cache for them.

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。