airbytehq / airbytehq/PyAirbyte

💡 Feature Proposal: Add a way to clean up after a lazy `Source.get_records()` generator is abandoned

未关闭
#442 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
344
派生
77
平均合并
1 天 11 小时
30 天内合并 PR
35

描述

Today, the `Source.get_records()` method returns a `LazyDataset` that can be iterated upon to get records.

Given a source declared like this:

```python
import airbyte as ab

source = ab.get_source(...)
```

You can iterate over records lazily like this:

```python
max_records = 10

dataset: ab.LazyDataset = source.get_records("my_stream")
for record, x in enumerate(dataset):
if x > max_records:
break
print(record)
```

This approach uses the dataset as an iterator and then aborts after the necessary count of records is found.

However, the connector process itself is not shut down when we stop iterating from it.

## Improvement Proposal

Ideally, we'd add a callback to close the connection on the lazy dataset - _and/or_ we'd operate like a context manager and auto-clean up the process when the context manager exits.

In practice, this has not caused any problems for our use cases - but it would be good to improve handling here.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。