airbytehq / airbytehq/PyAirbyte

Proposal: Add native `len()` support for the return of `get_records()` (`LazyDataset`)

Đang mở
#26 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
344
Fork
77
Merge trung bình
1 ngày 11 giờ
Pull request đã merge (30 ngày)
35

Mô tả

We because to implement this but then reverted because the `len()` implementation was exhausting the iterator and then the records were not iteratable after running `len()` on the result.

A workaround is to wrap in `len(list())`, and in this case the result will be converted to a list and then counted.

Design challenge:

There's an argument that we _should not_ directly implement len() over get_records() because it will carry the full cost of streaming all records, while discarding all records. If the user then wants to actually view the records, they will pay the cost of extracting records twice - and the results (including counts) could change between calls.

Logging this issue so that we can discuss and iterate on path forward. If we don't support directly, perhaps we can provide better guidance to users in a better error message.

Note:

The `len()` call _is_ directly supported on CachedDataset objects, and there's no similar issue with performance. We have already optimized that codepath to use `count()` in SQL to provide optimized `len()` results.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.