airbytehq / airbytehq/airbyte

[source-klaviyo] Missing data due to delay in attribution data

Đang mở
#61,001 3 bình luận 1 reaction 0 người được giao Xem trên GitHub
area/connectors community connectors/source/klaviyo team/extensibility type/bug
Ngôn ngữ chính
Python
Star
22.1k
Fork
5.4k
Merge trung bình
5 giờ
Pull request đã merge (30 ngày)
671

Mô tả

### Connector Name

source-klaviyo

### Connector Version

2.14.11

### What step the error happened?

During the sync

### Relevant information

Request & Bug: Configurable Lookback Window for Incremental Syncs (to Handle Delayed Attribution in Klaviyo Events)


Context:

Klaviyo events such as purchases often have attribution (e.g., campaign source). Currently, this data is added to the event instantly, and for some events there is an delay in adding this data. However, in the near future, this will have a delay of 3 hours for all events with attribution data. Klaviyo events are immutable and do not have an updated_at field. The main timestamp available (datetime) reflects when the event occurred, not when attribution or other enrichment is added.


This causes Airbyte’s incremental syncs—which use datetime as the cursor—to miss important attribution data that arrives after the initial event is extracted. See Klaviyo docs:



The problem:




  • When events are enriched after creation (e.g., attribution added 3h after purchase), their datetime does not change.




  • Incremental syncs based on datetime will not re-sync these updated events.




  • There is no updated_at or similar field to use for an accurate cursor.




Proposed Solution: Configurable Lookback Window




  • Add a user-configurable “lookback window” to the Klaviyo source connector. Preferably, only for specific user defined events.




  • On every incremental sync, re-fetch all events where datetime is within N hours before the last synced timestamp.




    • Example: If last sync fetched up to 2024-05-30T10:00:00Z, next sync would fetch from 2024-05-30T04:00:00Z onward (with a 6h window).






  • UI/Config:




    • Expose the window as a setting in the Airbyte connector config, with a safe default (e.g., 6 hours).




    • Update documentation to warn users about delayed attribution and recommend enabling this setting for full attribution data.






Benefits:




  • Prevents data loss due to late-arriving or enriched events.




  • Flexible—users can adjust the window based on observed data patterns.



### Relevant log output

```shell

```

### Contribute

- [ ] Yes, I want to contribute

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.