airbytehq / airbytehq/airbyte-python-cdk

OffsetIncrement.next_page_token re-extracts the full page to count records, discarding the retriever's last_page_size

オープン
#1,136 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
community
主要言語
Python
スター
26
フォーク
53
平均マージ
2日 6時間
マージ済み PR(30日)
10

説明

## Symptom

Every declarative stream paginated with `DefaultPaginator` + `OffsetIncrement` parses and extracts each response page twice: once in the retriever (record emission) and a second time inside the pagination strategy, purely to count records.

## Root cause

At airbyte-cdk 7.3.1, `airbyte_cdk/sources/declarative/requesters/paginators/strategies/offset_increment.py` (`next_page_token`, lines ~78-92):

```python
decoded_response = next(self.decoder.decode(response))

if self.extractor:
page_size_from_response = len(list(self.extractor.extract_records(response=response)))
last_page_size = (
page_size_from_response if page_size_from_response is not None else last_page_size
)
```

- `self.extractor.extract_records(response=response)` re-runs the full record extraction and materializes the whole page into a list just to take `len()`.
- The wiring is unconditional for declarative sources: `model_to_component_factory.py` passes `extractor_model=model.record_selector.extractor` into `create_default_paginator` >> `create_offset_increment`, so the paginator always holds a second instance of the stream's extractor.
- The incoming `last_page_size` argument - already counted per emitted record by `SimpleRetriever` - is effectively always overwritten: `len()` never returns `None`, so the fallback branch is dead.

## Impact

Double CPU and a transient second in-memory copy of every page on the pagination hot path. For large-page APIs this is material: source-google-analytics-data-api pages are up to 100,000 rows / ~45 MB JSON per page, and the same connector's extraction path was measured around 1.17 GiB RSS per 100k-row page during incident triage - the counting pass adds a second full materialization on top.

## Reproduction

Any manifest with `DefaultPaginator` + `OffsetIncrement`; add a counter or log line to the record extractor - it fires twice per page (once from the retriever, once from `next_page_token`).

## Suggestion

Count from what the retriever already extracted (it computes `last_page_size` per record) instead of re-parsing the raw response, or make the extractor-based counting opt-in for the nested-record cases it was added for.

## Precedent

Found while reviewing https://github.com/airbytehq/airbyte/pull/83188 and https://github.com/airbytehq/airbyte/pull/83318 (GA4 pagination fix: `CursorPagination` >> `OffsetIncrement` on 25k-100k row pages).

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

Start in airbyte_cdk/sources/declarative/requesters/paginators/strategies/offset_increment.py at next_page_token, then trace last_page_size from SimpleRetriever and the extractor wiring in model_to_component_factory.py. Compare the DefaultPaginator and OffsetIncrement paths, including nested-record cases. Done means pagination no longer re-extracts ordinary pages while record counts remain correct.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
python
領域
api, backend
issue の種類
バグ
難易度
3/5
見積もり時間
1〜2日
活発さ
活発
明瞭さ
おおむね明確
初心者へのやさしさ
64/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。