airbytehq / airbytehq/PyAirbyte

Race condition: missing stream state

Đang mở
#703 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
344
Fork
77
Merge trung bình
1 ngày 11 giờ
Pull request đã merge (30 ngày)
35

Mô tả

Hi! 👋🏻
We're facing a race condition about twice a week, where we query `_airbyte_state` and the state of a stream is missing from that table.

### Explanation
Our investigation led to this [session.commit()](https://github.com/airbytehq/PyAirbyte/blob/68078b6864f63575c82dffa1adec82ec741fcc92/airbyte/caches/_state_backend.py#L174).
At the end of a synchronization, two queries are sent:
1. Delete the state of the stream
2. Insert the new state.

As it is not happening in a single transaction, it is possible to try to read `_airbyte_state` during the short time the state has been deleted, and before the new one is inserted.

---

### Context
Our infrastructure may run some very specific streams concurrently. They are configured to do a `full_refresh` and use the `append` write strategy, and concurrent run are fine from our perspective.
Before a stream runs, we check the `_airbyte_state` table to detect if the state already exists. When there's no state, we try to automatically migrate the stream from Airbyte to PyAirbyte.

We stumbled upon cases where we try to read the state after PyAirbyte deleted a state and before it inserted the new one.
Using a Snowflake cache is certainly part of what highlights the race condition, as queries can be queued there.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.