airbytehq / airbytehq/PyAirbyte

💡 Proposal: `get_state()` and `set_state()` on the Cache interface

未关闭
#680 0 条评论 0 个 reaction 已指派 0 人 已被 @benjaminwilen 认领 在 GitHub 查看
主要语言
Python
星标
344
派生
77
平均合并
1 天 11 小时
30 天内合并 PR
35

描述

This proposal would give a public API for reading and writing stream state for a `Cache` interface.

Signature might be something like:

```py
class CacheBase:
...

def get_state(
stream_name: str,
) -> StreamState | None:
"""Return a stream state object for the provided stream name.

This is a thin wrapper around the internal `StateProvider` interface.
"""
...

def set_state(
stream_name: str,
stream_state: StreamState | dict,
) -> None:
"""Set a stream state object for the provided stream name.

This is a thin wrapper around the internal `StateWriter` interface.
"""
...

def migrate_state(
streams: list[str] | Literal["*"],
to_cache: CacheBase,
) -> None:
"""Copies all matching stream states to the specified `Cache` object.

This is a thin wrapper around the respective `Cache` objects'
`get_state` and `set_state` methods.
"""
...
```

Other considerations:

1. Since there are as of now many different strongly typed `State` classes (`StateMessage`, `StreamState`, etc.), we'd want to think carefully on which (if any) we feel comfortable to make a part of the public interface.
2. It might actually be cleaner to get/set values as `dict` objects, since that would fully avoid needing to expose a public API for the state object itself. (Tradeoff: `dict` objects may be difficult to parse.)
3. It should be noted that manually modifying a `state` artifact for a stream basically _always_ "voids the warranty", and sources and not guaranteed to have stable `state` artifact interfaces over time.
5. This feature is actually more appropriate for migration of state - such as during renames or moving from one state backend to another, or from one table name/alias to a new one. For this reason, I've included a possible `migrate_state()` method in the above which could streamline a "copy-all"-type operation.
6. Internally, states are often cached in memory by the `StateProvider` class. We'd need to take care to invalidate or refresh the caches after a "set" action.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。