airbytehq / airbytehq/airbyte-python-cdk

Improve error message when a persisted cursor value cannot be parsed (ConcurrentPerPartitionCursor._set_global_state)

Open
#1,120 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
26
Forks
53
Avg merge
2d 6h
Merged PRs (30d)
10

Description

## Problem

When a persisted cursor state value cannot be parsed with the stream's configured `cursor_datetime_formats`, `ConcurrentPerPartitionCursor._set_global_state()` lets the raw `ValueError` escape during stream construction. The sync dies before any record is read, and the user-facing failure is:

```
No format in ['%s', '%s'] matching 1698686995.0
```

Observed in production on a manifest-only source (`source-slack`, `threads` stream) where the stored global cursor value carried a fractional epoch (`"1698686995.0"`) while the stream declares `cursor_datetime_formats: ["%s"]`.

## Why the current message is poor

- Leaks implementation details (`%s` format tokens, an internal Python list) with no user-meaningful vocabulary.
- Does not say which stream, which cursor field, or that the value came from saved sync state.
- Offers no remediation (e.g. "refresh/clear the stream state").
- Classified as `system_error` / `failure_origin: source`, so it looks like a connector crash rather than an unparseable-state condition.
- Duplicate entries in the format list (`['%s', '%s']`) because `CustomFormatConcurrentStreamStateConverter.__init__` appends `datetime_format` to `input_datetime_formats`, which makes the message look nonsensical.

## Suggested improvements

1. Wrap the failure in an `AirbyteTracedException` with a `config_error`-style (non-`system_error`) failure type, naming the stream, the cursor field, the offending value, and the accepted formats, plus a remediation hint pointing at refreshing the stream's state.
2. Consider mirroring the tolerance already added for record cursor values in [#758](https://github.com/airbytehq/airbyte-python-cdk/pull/758) (`observe()` logs a warning and skips instead of raising) so a single unparseable *state* value degrades to a warning rather than failing the sync.
3. De-duplicate the formats reported in the message.

## References

- Raise site: `airbyte_cdk/sources/streams/concurrent/state_converters/datetime_stream_state_converter.py` (`CustomFormatConcurrentStreamStateConverter.parse_timestamp`)
- Call path: `ConcurrentPerPartitionCursor._set_initial_state` → `_set_global_state` → `parse_value`
- Prior art for tolerant handling: https://github.com/airbytehq/airbyte-python-cdk/pull/758
- Originating oncall issue: https://github.com/airbytehq/oncall/issues/13342

---
[Devin session](https://app.devin.ai/sessions/e76104a6da8443f48a2ff8c5748df5b4)

Contributor guide

Open the contributing guide

Research direction

Start in airbyte_cdk/sources/streams/concurrent/state_converters/datetime_stream_state_converter.py, especially CustomFormatConcurrentStreamStateConverter.parse_timestamp, then follow ConcurrentPerPartitionCursor._set_initial_state → _set_global_state → parse_value. Compare the handling with observe() and prior art in #758. Done means unparseable persisted state produces the agreed user-facing outcome, with relevant stream, cursor, value, formats, and remediation context without duplicate formats.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.