airbytehq / airbytehq/airbyte-python-cdk
Legacy Stream path: checkpoint reader and state assignment ignore the configured sync mode - cursor-bearing streams leak and consume cursor state under full refresh
- Ngôn ngữ chính
- Python
- Star
- 26
- Fork
- 53
- Merge trung bình
- 2 ngày 6 giờ
- Pull request đã merge (30 ngày)
- 10
Mô tả
## Symptom
A legacy (non-declarative) `Stream` that declares a `cursor_field` behaves incrementally even when the connection configures the stream as `full_refresh`:
1. it EMITS real cursor state during a full-refresh sync, and
2. it CONSUMES persisted cursor state during a full-refresh sync - the read comes back filtered.
Together these make a "full refresh" silently partial: a mid-job retry attempt resumes from the cursor emitted by the first attempt, and a connection switched from incremental back to full refresh keeps filtering by the stale cursor.
## Root cause (refs at v7.17.4)
- `airbyte_cdk/sources/abstract_source.py` L259-263: `stream_instance.state = ` is assigned whenever the platform provides state - the configured sync mode is never consulted.
- `airbyte_cdk/sources/streams/core.py` L526-533 (`_checkpoint_mode`): returns `CheckpointMode.INCREMENTAL` whenever `is_resumable` and `cursor_field` are set - again independent of the configured sync mode.
- `airbyte_cdk/sources/streams/core.py` L344-345: `is_resumable` short-circuits to `True` when `supports_incremental` is `True` (i.e. whenever `cursor_field` is non-empty).
Consequence: the `FullRefreshCheckpointReader` (whose `__ab_no_cursor_state_message` marker is deliberately discarded on read-back, `abstract_source.py` L258-259) is bypassed for ANY cursor-bearing stream, even on a full-refresh-configured connection. The state a full-refresh sync emits is a normal cursor blob that nothing discards later.
## Minimal reproduction
Take any legacy `Stream` subclass with a `cursor_field`, a `state` property and a state-driven request filter. Configure the stream as `full_refresh` in the catalog and run `read` twice, feeding the first run's emitted state into the second: run 1 emits a real cursor; run 2 issues filtered requests and returns a subset. Repeated end to end in airbytehq/airbyte#83704 with `source-facebook-marketing` (`ad_creatives_from_ads`): the pre-fix regression run shows the stream emitting cursor state where `6.0.2` emits only `__ab_no_cursor_state_message`, and a captured `request_params` confirms the cursor filter is applied under full refresh once state exists.
## Impact
Every legacy connector that adds incremental support to an existing stream must hand-gate BOTH directions:
- the state-write path (do not set `self.state` unless `sync_mode == SyncMode.incremental`), and
- the state-read path (drop the slice's persisted state for non-incremental runs).
Miss either one and full-refresh connections are exposed to silent partial reads after a retry or a sync-mode switch. Neither requirement is documented.
## Suggested direction
Either consult the configured sync mode in `_checkpoint_mode` / the state assignment in `AbstractSource._read_stream`, or document the required connector-side double gate for legacy streams. The declarative/concurrent paths are unaffected; this is specifically the legacy `Stream` plumbing.
## Precedent
Found during review of airbytehq/airbyte#83704 (source-facebook-marketing 6.1.0, making `ad_creatives_from_ads` incremental). The connector-level fixes and evidence: https://github.com/airbytehq/airbyte/pull/83704#issuecomment-5253236410 (write-side gate, regression evidence) and the follow-up commit gating the read side (`80b78957`).
Hướng dẫn đóng góp
Hướng nghiên cứu
Read the cited sections of airbyte_cdk/sources/abstract_source.py and airbyte_cdk/sources/streams/core.py, starting with _read_stream and _checkpoint_mode. Reproduce the issue with a cursor-bearing legacy Stream by running read twice and feeding the first run's state into the second. Done means full-refresh runs neither consume persisted cursor state nor emit a real cursor, while incremental behavior remains unchanged.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- python
- Lĩnh vực
- backend
- Loại issue
- Lỗi
- Độ khó
- 4/5
- Thời gian dự kiến
- 3-5 ngày
- Mức độ hoạt động
- Ít trao đổi
- Độ rõ ràng
- Khá rõ ràng
- Mức phù hợp với người mới
- 55/100