airbytehq / airbytehq/airbyte-python-cdk

Legacy Stream path: checkpoint reader and state assignment ignore the configured sync mode - cursor-bearing streams leak and consume cursor state under full refresh

Open
#1,112 0 comments 0 reactions 0 assignees View on GitHub
community
Dominant language
Python
Stars
26
Forks
53
Avg merge
2d 6h
Merged PRs (30d)
10

Description

## Symptom

A legacy (non-declarative) `Stream` that declares a `cursor_field` behaves incrementally even when the connection configures the stream as `full_refresh`:

1. it EMITS real cursor state during a full-refresh sync, and
2. it CONSUMES persisted cursor state during a full-refresh sync - the read comes back filtered.

Together these make a "full refresh" silently partial: a mid-job retry attempt resumes from the cursor emitted by the first attempt, and a connection switched from incremental back to full refresh keeps filtering by the stale cursor.

## Root cause (refs at v7.17.4)

- `airbyte_cdk/sources/abstract_source.py` L259-263: `stream_instance.state = ` is assigned whenever the platform provides state - the configured sync mode is never consulted.
- `airbyte_cdk/sources/streams/core.py` L526-533 (`_checkpoint_mode`): returns `CheckpointMode.INCREMENTAL` whenever `is_resumable` and `cursor_field` are set - again independent of the configured sync mode.
- `airbyte_cdk/sources/streams/core.py` L344-345: `is_resumable` short-circuits to `True` when `supports_incremental` is `True` (i.e. whenever `cursor_field` is non-empty).

Consequence: the `FullRefreshCheckpointReader` (whose `__ab_no_cursor_state_message` marker is deliberately discarded on read-back, `abstract_source.py` L258-259) is bypassed for ANY cursor-bearing stream, even on a full-refresh-configured connection. The state a full-refresh sync emits is a normal cursor blob that nothing discards later.

## Minimal reproduction

Take any legacy `Stream` subclass with a `cursor_field`, a `state` property and a state-driven request filter. Configure the stream as `full_refresh` in the catalog and run `read` twice, feeding the first run's emitted state into the second: run 1 emits a real cursor; run 2 issues filtered requests and returns a subset. Repeated end to end in airbytehq/airbyte#83704 with `source-facebook-marketing` (`ad_creatives_from_ads`): the pre-fix regression run shows the stream emitting cursor state where `6.0.2` emits only `__ab_no_cursor_state_message`, and a captured `request_params` confirms the cursor filter is applied under full refresh once state exists.

## Impact

Every legacy connector that adds incremental support to an existing stream must hand-gate BOTH directions:

- the state-write path (do not set `self.state` unless `sync_mode == SyncMode.incremental`), and
- the state-read path (drop the slice's persisted state for non-incremental runs).

Miss either one and full-refresh connections are exposed to silent partial reads after a retry or a sync-mode switch. Neither requirement is documented.

## Suggested direction

Either consult the configured sync mode in `_checkpoint_mode` / the state assignment in `AbstractSource._read_stream`, or document the required connector-side double gate for legacy streams. The declarative/concurrent paths are unaffected; this is specifically the legacy `Stream` plumbing.

## Precedent

Found during review of airbytehq/airbyte#83704 (source-facebook-marketing 6.1.0, making `ad_creatives_from_ads` incremental). The connector-level fixes and evidence: https://github.com/airbytehq/airbyte/pull/83704#issuecomment-5253236410 (write-side gate, regression evidence) and the follow-up commit gating the read side (`80b78957`).

Contributor guide

Open the contributing guide

Research direction

Read the cited sections of airbyte_cdk/sources/abstract_source.py and airbyte_cdk/sources/streams/core.py, starting with _read_stream and _checkpoint_mode. Reproduce the issue with a cursor-bearing legacy Stream by running read twice and feeding the first run's state into the second. Done means full-refresh runs neither consume persisted cursor state nor emit a real cursor, while incremental behavior remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.