airbytehq / airbytehq/airbyte-python-cdk

Legacy Stream path: checkpoint reader and state assignment ignore the configured sync mode - cursor-bearing streams leak and consume cursor state under full refresh

Abierto
#1,112 0 comentarios 0 reacciones 0 asignados Ver en GitHub
community
Lenguaje dominante
Python
Estrellas
26
Forks
53
Merge medio
2 d 6 h
PR fusionados (30 d)
10

Descripción

## Symptom

A legacy (non-declarative) `Stream` that declares a `cursor_field` behaves incrementally even when the connection configures the stream as `full_refresh`:

1. it EMITS real cursor state during a full-refresh sync, and
2. it CONSUMES persisted cursor state during a full-refresh sync - the read comes back filtered.

Together these make a "full refresh" silently partial: a mid-job retry attempt resumes from the cursor emitted by the first attempt, and a connection switched from incremental back to full refresh keeps filtering by the stale cursor.

## Root cause (refs at v7.17.4)

- `airbyte_cdk/sources/abstract_source.py` L259-263: `stream_instance.state = ` is assigned whenever the platform provides state - the configured sync mode is never consulted.
- `airbyte_cdk/sources/streams/core.py` L526-533 (`_checkpoint_mode`): returns `CheckpointMode.INCREMENTAL` whenever `is_resumable` and `cursor_field` are set - again independent of the configured sync mode.
- `airbyte_cdk/sources/streams/core.py` L344-345: `is_resumable` short-circuits to `True` when `supports_incremental` is `True` (i.e. whenever `cursor_field` is non-empty).

Consequence: the `FullRefreshCheckpointReader` (whose `__ab_no_cursor_state_message` marker is deliberately discarded on read-back, `abstract_source.py` L258-259) is bypassed for ANY cursor-bearing stream, even on a full-refresh-configured connection. The state a full-refresh sync emits is a normal cursor blob that nothing discards later.

## Minimal reproduction

Take any legacy `Stream` subclass with a `cursor_field`, a `state` property and a state-driven request filter. Configure the stream as `full_refresh` in the catalog and run `read` twice, feeding the first run's emitted state into the second: run 1 emits a real cursor; run 2 issues filtered requests and returns a subset. Repeated end to end in airbytehq/airbyte#83704 with `source-facebook-marketing` (`ad_creatives_from_ads`): the pre-fix regression run shows the stream emitting cursor state where `6.0.2` emits only `__ab_no_cursor_state_message`, and a captured `request_params` confirms the cursor filter is applied under full refresh once state exists.

## Impact

Every legacy connector that adds incremental support to an existing stream must hand-gate BOTH directions:

- the state-write path (do not set `self.state` unless `sync_mode == SyncMode.incremental`), and
- the state-read path (drop the slice's persisted state for non-incremental runs).

Miss either one and full-refresh connections are exposed to silent partial reads after a retry or a sync-mode switch. Neither requirement is documented.

## Suggested direction

Either consult the configured sync mode in `_checkpoint_mode` / the state assignment in `AbstractSource._read_stream`, or document the required connector-side double gate for legacy streams. The declarative/concurrent paths are unaffected; this is specifically the legacy `Stream` plumbing.

## Precedent

Found during review of airbytehq/airbyte#83704 (source-facebook-marketing 6.1.0, making `ad_creatives_from_ads` incremental). The connector-level fixes and evidence: https://github.com/airbytehq/airbyte/pull/83704#issuecomment-5253236410 (write-side gate, regression evidence) and the follow-up commit gating the read side (`80b78957`).

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Read the cited sections of airbyte_cdk/sources/abstract_source.py and airbyte_cdk/sources/streams/core.py, starting with _read_stream and _checkpoint_mode. Reproduce the issue with a cursor-bearing legacy Stream by running read twice and feeding the first run's state into the second. Done means full-refresh runs neither consume persisted cursor state nor emit a real cursor, while incremental behavior remains unchanged.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
python
Área
backend
Tipo de issue
Error
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Tranquilo
Claridad
Bastante claro
Aptitud para principiantes
55/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.