airbytehq / airbytehq/airbyte-python-cdk

Improve error message when a persisted cursor value cannot be parsed (ConcurrentPerPartitionCursor._set_global_state)

未关闭
#1,120 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
enhancement
主要语言
Python
星标
26
派生
53
平均合并
2 天 6 小时
30 天内合并 PR
10

描述

## Problem

When a persisted cursor state value cannot be parsed with the stream's configured `cursor_datetime_formats`, `ConcurrentPerPartitionCursor._set_global_state()` lets the raw `ValueError` escape during stream construction. The sync dies before any record is read, and the user-facing failure is:

```
No format in ['%s', '%s'] matching 1698686995.0
```

Observed in production on a manifest-only source (`source-slack`, `threads` stream) where the stored global cursor value carried a fractional epoch (`"1698686995.0"`) while the stream declares `cursor_datetime_formats: ["%s"]`.

## Why the current message is poor

- Leaks implementation details (`%s` format tokens, an internal Python list) with no user-meaningful vocabulary.
- Does not say which stream, which cursor field, or that the value came from saved sync state.
- Offers no remediation (e.g. "refresh/clear the stream state").
- Classified as `system_error` / `failure_origin: source`, so it looks like a connector crash rather than an unparseable-state condition.
- Duplicate entries in the format list (`['%s', '%s']`) because `CustomFormatConcurrentStreamStateConverter.__init__` appends `datetime_format` to `input_datetime_formats`, which makes the message look nonsensical.

## Suggested improvements

1. Wrap the failure in an `AirbyteTracedException` with a `config_error`-style (non-`system_error`) failure type, naming the stream, the cursor field, the offending value, and the accepted formats, plus a remediation hint pointing at refreshing the stream's state.
2. Consider mirroring the tolerance already added for record cursor values in [#758](https://github.com/airbytehq/airbyte-python-cdk/pull/758) (`observe()` logs a warning and skips instead of raising) so a single unparseable *state* value degrades to a warning rather than failing the sync.
3. De-duplicate the formats reported in the message.

## References

- Raise site: `airbyte_cdk/sources/streams/concurrent/state_converters/datetime_stream_state_converter.py` (`CustomFormatConcurrentStreamStateConverter.parse_timestamp`)
- Call path: `ConcurrentPerPartitionCursor._set_initial_state` → `_set_global_state` → `parse_value`
- Prior art for tolerant handling: https://github.com/airbytehq/airbyte-python-cdk/pull/758
- Originating oncall issue: https://github.com/airbytehq/oncall/issues/13342

---
[Devin session](https://app.devin.ai/sessions/e76104a6da8443f48a2ff8c5748df5b4)

贡献指南

打开贡献指南

调研方向

Start in airbyte_cdk/sources/streams/concurrent/state_converters/datetime_stream_state_converter.py, especially CustomFormatConcurrentStreamStateConverter.parse_timestamp, then follow ConcurrentPerPartitionCursor._set_initial_state → _set_global_state → parse_value. Compare the handling with observe() and prior art in #758. Done means unparseable persisted state produces the agreed user-facing outcome, with relevant stream, cursor, value, formats, and remediation context without duplicate formats.

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
backend
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
活跃
描述清晰度
基本清楚
新手友好度
48/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。