airbytehq / airbytehq/airbyte-python-cdk
Improve error message when a persisted cursor value cannot be parsed (ConcurrentPerPartitionCursor._set_global_state)
- Vorherrschende Sprache
- Python
- Sterne
- 26
- Forks
- 53
- Ø Merge
- 2 T. 6 Std.
- Gemergte PRs (30 T.)
- 10
Beschreibung
## Problem
When a persisted cursor state value cannot be parsed with the stream's configured `cursor_datetime_formats`, `ConcurrentPerPartitionCursor._set_global_state()` lets the raw `ValueError` escape during stream construction. The sync dies before any record is read, and the user-facing failure is:
```
No format in ['%s', '%s'] matching 1698686995.0
```
Observed in production on a manifest-only source (`source-slack`, `threads` stream) where the stored global cursor value carried a fractional epoch (`"1698686995.0"`) while the stream declares `cursor_datetime_formats: ["%s"]`.
## Why the current message is poor
- Leaks implementation details (`%s` format tokens, an internal Python list) with no user-meaningful vocabulary.
- Does not say which stream, which cursor field, or that the value came from saved sync state.
- Offers no remediation (e.g. "refresh/clear the stream state").
- Classified as `system_error` / `failure_origin: source`, so it looks like a connector crash rather than an unparseable-state condition.
- Duplicate entries in the format list (`['%s', '%s']`) because `CustomFormatConcurrentStreamStateConverter.__init__` appends `datetime_format` to `input_datetime_formats`, which makes the message look nonsensical.
## Suggested improvements
1. Wrap the failure in an `AirbyteTracedException` with a `config_error`-style (non-`system_error`) failure type, naming the stream, the cursor field, the offending value, and the accepted formats, plus a remediation hint pointing at refreshing the stream's state.
2. Consider mirroring the tolerance already added for record cursor values in [#758](https://github.com/airbytehq/airbyte-python-cdk/pull/758) (`observe()` logs a warning and skips instead of raising) so a single unparseable *state* value degrades to a warning rather than failing the sync.
3. De-duplicate the formats reported in the message.
## References
- Raise site: `airbyte_cdk/sources/streams/concurrent/state_converters/datetime_stream_state_converter.py` (`CustomFormatConcurrentStreamStateConverter.parse_timestamp`)
- Call path: `ConcurrentPerPartitionCursor._set_initial_state` → `_set_global_state` → `parse_value`
- Prior art for tolerant handling: https://github.com/airbytehq/airbyte-python-cdk/pull/758
- Originating oncall issue: https://github.com/airbytehq/oncall/issues/13342
---
[Devin session](https://app.devin.ai/sessions/e76104a6da8443f48a2ff8c5748df5b4)
Beitragsleitfaden
Rechercherichtung
Start in airbyte_cdk/sources/streams/concurrent/state_converters/datetime_stream_state_converter.py, especially CustomFormatConcurrentStreamStateConverter.parse_timestamp, then follow ConcurrentPerPartitionCursor._set_initial_state → _set_global_state → parse_value. Compare the handling with observe() and prior art in #758. Done means unparseable persisted state produces the agreed user-facing outcome, with relevant stream, cursor, value, formats, and remediation context without duplicate formats.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- python
- Bereich
- backend
- Issue-Typ
- Bug
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Aktiv
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 48/100