airbytehq / airbytehq/PyAirbyte
Support aliased Destinations or Sources (for State Caches)
- 主要言語
- Python
- スター
- 344
- フォーク
- 77
- 平均マージ
- 1日 11時間
- マージ済み PR(30日)
- 35
説明
The current implementation of the [DestinationStreamStateModel](https://github.com/airbytehq/PyAirbyte/blob/f7b88eba400d7aa768c8c370cfbac6f18dfc61c6/airbyte/caches/_state_backend.py#L68) assumes a one-to-one relationship between a [Source](https://github.com/airbytehq/PyAirbyte/blob/f7b88eba400d7aa768c8c370cfbac6f18dfc61c6/airbyte/sources/base.py#L52) name and a [Destination](https://github.com/airbytehq/PyAirbyte/blob/f7b88eba400d7aa768c8c370cfbac6f18dfc61c6/airbyte/destinations/base.py#L38) name. This is fine in principle, under the assumption that the Connection only has one instance of the Source type. However, this is not always the case, lets imagine we have multiple cases of a Google Sheets Source, for which we want to keep its state tracked via `state_cache` when writing to a destination. How can we achieve this?
In Airbyte, we achieve this by aliasing the Sources and Destinations:


By separating the ORM model to write based on the aliased Source and Destination/Cache names, we can support multiple types of Sources and Destinations and still hold cache for them.
コントリビューションガイド
調査の方向性
Start with airbyte/caches/_state_backend.py and the referenced Source and Destination definitions in airbyte/sources/base.py and airbyte/destinations/base.py. Trace how DestinationStreamStateModel selects names for state storage, then define completion as distinct aliased source and destination/cache names supporting multiple instances without state collisions.
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- python
- 領域
- backend, data-engineering
- issue の種類
- 機能追加
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 活発さ
- 停滞
- 明瞭さ
- おおむね明確
- 初心者へのやさしさ
- 35/100