[source-klaviyo] Missing data due to delay in attribution data
- Ngôn ngữ chính
- Python
- Star
- 22.1k
- Fork
- 5.4k
- Merge trung bình
- 5 giờ
- Pull request đã merge (30 ngày)
- 671
Mô tả
### Connector Name
source-klaviyo
### Connector Version
2.14.11
### What step the error happened?
During the sync
### Relevant information
Request & Bug: Configurable Lookback Window for Incremental Syncs (to Handle Delayed Attribution in Klaviyo Events)
Context:
Klaviyo events such as purchases often have attribution (e.g., campaign source). Currently, this data is added to the event instantly, and for some events there is an delay in adding this data. However, in the near future, this will have a delay of 3 hours for all events with attribution data. Klaviyo events are immutable and do not have an updated_at field. The main timestamp available (datetime) reflects when the event occurred, not when attribution or other enrichment is added.
This causes Airbyte’s incremental syncs—which use datetime as the cursor—to miss important attribution data that arrives after the initial event is extracted. See Klaviyo docs:
The problem:
-
When events are enriched after creation (e.g., attribution added 3h after purchase), their
datetimedoes not change. -
Incremental syncs based on
datetimewill not re-sync these updated events. -
There is no
updated_ator similar field to use for an accurate cursor.
Proposed Solution: Configurable Lookback Window
-
Add a user-configurable “lookback window” to the Klaviyo source connector. Preferably, only for specific user defined events.
-
On every incremental sync, re-fetch all events where
datetimeis within N hours before the last synced timestamp.-
Example: If last sync fetched up to
2024-05-30T10:00:00Z, next sync would fetch from2024-05-30T04:00:00Zonward (with a 6h window).
-
-
UI/Config:
-
Expose the window as a setting in the Airbyte connector config, with a safe default (e.g., 6 hours).
-
Update documentation to warn users about delayed attribution and recommend enabling this setting for full attribution data.
-
Benefits:
-
Prevents data loss due to late-arriving or enriched events.
-
Flexible—users can adjust the window based on observed data patterns.
### Relevant log output
```shell
```
### Contribute
- [ ] Yes, I want to contribute
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.