airbytehq / airbytehq/airbyte-python-cdk

Legacy Stream path: checkpoint reader and state assignment ignore the configured sync mode - cursor-bearing streams leak and consume cursor state under full refresh

Đang mở
#1,112 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
community
Ngôn ngữ chính
Python
Star
26
Fork
53
Merge trung bình
2 ngày 6 giờ
Pull request đã merge (30 ngày)
10

Mô tả

## Symptom

A legacy (non-declarative) `Stream` that declares a `cursor_field` behaves incrementally even when the connection configures the stream as `full_refresh`:

1. it EMITS real cursor state during a full-refresh sync, and
2. it CONSUMES persisted cursor state during a full-refresh sync - the read comes back filtered.

Together these make a "full refresh" silently partial: a mid-job retry attempt resumes from the cursor emitted by the first attempt, and a connection switched from incremental back to full refresh keeps filtering by the stale cursor.

## Root cause (refs at v7.17.4)

- `airbyte_cdk/sources/abstract_source.py` L259-263: `stream_instance.state = ` is assigned whenever the platform provides state - the configured sync mode is never consulted.
- `airbyte_cdk/sources/streams/core.py` L526-533 (`_checkpoint_mode`): returns `CheckpointMode.INCREMENTAL` whenever `is_resumable` and `cursor_field` are set - again independent of the configured sync mode.
- `airbyte_cdk/sources/streams/core.py` L344-345: `is_resumable` short-circuits to `True` when `supports_incremental` is `True` (i.e. whenever `cursor_field` is non-empty).

Consequence: the `FullRefreshCheckpointReader` (whose `__ab_no_cursor_state_message` marker is deliberately discarded on read-back, `abstract_source.py` L258-259) is bypassed for ANY cursor-bearing stream, even on a full-refresh-configured connection. The state a full-refresh sync emits is a normal cursor blob that nothing discards later.

## Minimal reproduction

Take any legacy `Stream` subclass with a `cursor_field`, a `state` property and a state-driven request filter. Configure the stream as `full_refresh` in the catalog and run `read` twice, feeding the first run's emitted state into the second: run 1 emits a real cursor; run 2 issues filtered requests and returns a subset. Repeated end to end in airbytehq/airbyte#83704 with `source-facebook-marketing` (`ad_creatives_from_ads`): the pre-fix regression run shows the stream emitting cursor state where `6.0.2` emits only `__ab_no_cursor_state_message`, and a captured `request_params` confirms the cursor filter is applied under full refresh once state exists.

## Impact

Every legacy connector that adds incremental support to an existing stream must hand-gate BOTH directions:

- the state-write path (do not set `self.state` unless `sync_mode == SyncMode.incremental`), and
- the state-read path (drop the slice's persisted state for non-incremental runs).

Miss either one and full-refresh connections are exposed to silent partial reads after a retry or a sync-mode switch. Neither requirement is documented.

## Suggested direction

Either consult the configured sync mode in `_checkpoint_mode` / the state assignment in `AbstractSource._read_stream`, or document the required connector-side double gate for legacy streams. The declarative/concurrent paths are unaffected; this is specifically the legacy `Stream` plumbing.

## Precedent

Found during review of airbytehq/airbyte#83704 (source-facebook-marketing 6.1.0, making `ad_creatives_from_ads` incremental). The connector-level fixes and evidence: https://github.com/airbytehq/airbyte/pull/83704#issuecomment-5253236410 (write-side gate, regression evidence) and the follow-up commit gating the read side (`80b78957`).

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Read the cited sections of airbyte_cdk/sources/abstract_source.py and airbyte_cdk/sources/streams/core.py, starting with _read_stream and _checkpoint_mode. Reproduce the issue with a cursor-bearing legacy Stream by running read twice and feeding the first run's state into the second. Done means full-refresh runs neither consume persisted cursor state nor emit a real cursor, while incremental behavior remains unchanged.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
python
Lĩnh vực
backend
Loại issue
Lỗi
Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức độ hoạt động
Ít trao đổi
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
55/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.