airbytehq / airbytehq/PyAirbyte

Support aliased Destinations or Sources (for State Caches)

Đang mở
#570 3 bình luận 0 reaction 0 người được giao Xem trên GitHub
accepting pull requests
Ngôn ngữ chính
Python
Star
344
Fork
77
Merge trung bình
1 ngày 11 giờ
Pull request đã merge (30 ngày)
35

Mô tả

The current implementation of the [DestinationStreamStateModel](https://github.com/airbytehq/PyAirbyte/blob/f7b88eba400d7aa768c8c370cfbac6f18dfc61c6/airbyte/caches/_state_backend.py#L68) assumes a one-to-one relationship between a [Source](https://github.com/airbytehq/PyAirbyte/blob/f7b88eba400d7aa768c8c370cfbac6f18dfc61c6/airbyte/sources/base.py#L52) name and a [Destination](https://github.com/airbytehq/PyAirbyte/blob/f7b88eba400d7aa768c8c370cfbac6f18dfc61c6/airbyte/destinations/base.py#L38) name. This is fine in principle, under the assumption that the Connection only has one instance of the Source type. However, this is not always the case, lets imagine we have multiple cases of a Google Sheets Source, for which we want to keep its state tracked via `state_cache` when writing to a destination. How can we achieve this?

In Airbyte, we achieve this by aliasing the Sources and Destinations:
![Image](https://github.com/user-attachments/assets/284ad34a-ccc3-4160-9ace-6466cf605e39)
![Image](https://github.com/user-attachments/assets/39809b3b-25a1-4013-8b81-712fda922947)

By separating the ORM model to write based on the aliased Source and Destination/Cache names, we can support multiple types of Sources and Destinations and still hold cache for them.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.