airbytehq / airbytehq/airbyte-python-cdk

OffsetIncrement.next_page_token re-extracts the full page to count records, discarding the retriever's last_page_size

Đang mở
#1,136 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
community
Ngôn ngữ chính
Python
Star
26
Fork
53
Merge trung bình
2 ngày 6 giờ
Pull request đã merge (30 ngày)
10

Mô tả

## Symptom

Every declarative stream paginated with `DefaultPaginator` + `OffsetIncrement` parses and extracts each response page twice: once in the retriever (record emission) and a second time inside the pagination strategy, purely to count records.

## Root cause

At airbyte-cdk 7.3.1, `airbyte_cdk/sources/declarative/requesters/paginators/strategies/offset_increment.py` (`next_page_token`, lines ~78-92):

```python
decoded_response = next(self.decoder.decode(response))

if self.extractor:
page_size_from_response = len(list(self.extractor.extract_records(response=response)))
last_page_size = (
page_size_from_response if page_size_from_response is not None else last_page_size
)
```

- `self.extractor.extract_records(response=response)` re-runs the full record extraction and materializes the whole page into a list just to take `len()`.
- The wiring is unconditional for declarative sources: `model_to_component_factory.py` passes `extractor_model=model.record_selector.extractor` into `create_default_paginator` >> `create_offset_increment`, so the paginator always holds a second instance of the stream's extractor.
- The incoming `last_page_size` argument - already counted per emitted record by `SimpleRetriever` - is effectively always overwritten: `len()` never returns `None`, so the fallback branch is dead.

## Impact

Double CPU and a transient second in-memory copy of every page on the pagination hot path. For large-page APIs this is material: source-google-analytics-data-api pages are up to 100,000 rows / ~45 MB JSON per page, and the same connector's extraction path was measured around 1.17 GiB RSS per 100k-row page during incident triage - the counting pass adds a second full materialization on top.

## Reproduction

Any manifest with `DefaultPaginator` + `OffsetIncrement`; add a counter or log line to the record extractor - it fires twice per page (once from the retriever, once from `next_page_token`).

## Suggestion

Count from what the retriever already extracted (it computes `last_page_size` per record) instead of re-parsing the raw response, or make the extractor-based counting opt-in for the nested-record cases it was added for.

## Precedent

Found while reviewing https://github.com/airbytehq/airbyte/pull/83188 and https://github.com/airbytehq/airbyte/pull/83318 (GA4 pagination fix: `CursorPagination` >> `OffsetIncrement` on 25k-100k row pages).

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Start in airbyte_cdk/sources/declarative/requesters/paginators/strategies/offset_increment.py at next_page_token, then trace last_page_size from SimpleRetriever and the extractor wiring in model_to_component_factory.py. Compare the DefaultPaginator and OffsetIncrement paths, including nested-record cases. Done means pagination no longer re-extracts ordinary pages while record counts remain correct.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
python
Lĩnh vực
api, backend
Loại issue
Lỗi
Độ khó
3/5
Thời gian dự kiến
1-2 ngày
Mức độ hoạt động
Sôi nổi
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
64/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.