airbytehq / airbytehq/airbyte-python-cdk

OffsetIncrement.next_page_token re-extracts the full page to count records, discarding the retriever's last_page_size

Abierto
#1,136 0 comentarios 0 reacciones 0 asignados Ver en GitHub
community
Lenguaje dominante
Python
Estrellas
26
Forks
53
Merge medio
2 d 6 h
PR fusionados (30 d)
10

Descripción

## Symptom

Every declarative stream paginated with `DefaultPaginator` + `OffsetIncrement` parses and extracts each response page twice: once in the retriever (record emission) and a second time inside the pagination strategy, purely to count records.

## Root cause

At airbyte-cdk 7.3.1, `airbyte_cdk/sources/declarative/requesters/paginators/strategies/offset_increment.py` (`next_page_token`, lines ~78-92):

```python
decoded_response = next(self.decoder.decode(response))

if self.extractor:
page_size_from_response = len(list(self.extractor.extract_records(response=response)))
last_page_size = (
page_size_from_response if page_size_from_response is not None else last_page_size
)
```

- `self.extractor.extract_records(response=response)` re-runs the full record extraction and materializes the whole page into a list just to take `len()`.
- The wiring is unconditional for declarative sources: `model_to_component_factory.py` passes `extractor_model=model.record_selector.extractor` into `create_default_paginator` >> `create_offset_increment`, so the paginator always holds a second instance of the stream's extractor.
- The incoming `last_page_size` argument - already counted per emitted record by `SimpleRetriever` - is effectively always overwritten: `len()` never returns `None`, so the fallback branch is dead.

## Impact

Double CPU and a transient second in-memory copy of every page on the pagination hot path. For large-page APIs this is material: source-google-analytics-data-api pages are up to 100,000 rows / ~45 MB JSON per page, and the same connector's extraction path was measured around 1.17 GiB RSS per 100k-row page during incident triage - the counting pass adds a second full materialization on top.

## Reproduction

Any manifest with `DefaultPaginator` + `OffsetIncrement`; add a counter or log line to the record extractor - it fires twice per page (once from the retriever, once from `next_page_token`).

## Suggestion

Count from what the retriever already extracted (it computes `last_page_size` per record) instead of re-parsing the raw response, or make the extractor-based counting opt-in for the nested-record cases it was added for.

## Precedent

Found while reviewing https://github.com/airbytehq/airbyte/pull/83188 and https://github.com/airbytehq/airbyte/pull/83318 (GA4 pagination fix: `CursorPagination` >> `OffsetIncrement` on 25k-100k row pages).

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Start in airbyte_cdk/sources/declarative/requesters/paginators/strategies/offset_increment.py at next_page_token, then trace last_page_size from SimpleRetriever and the extractor wiring in model_to_component_factory.py. Compare the DefaultPaginator and OffsetIncrement paths, including nested-record cases. Done means pagination no longer re-extracts ordinary pages while record counts remain correct.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
python
Área
api, backend
Tipo de issue
Error
Dificultad
3/5
Tiempo estimado
1-2 días
Estado de actividad
Activo
Claridad
Bastante claro
Aptitud para principiantes
64/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.