airbytehq / airbytehq/airbyte

[source-klaviyo] Missing data due to delay in attribution data

Ouverte
#61,001 3 commentaires 1 réaction 0 personnes assignées Voir sur GitHub
area/connectors community connectors/source/klaviyo team/extensibility type/bug
Langage dominant
Python
Étoiles
22.1k
Forks
5.4k
Merge moyen
5 h
PR mergées (30 j)
671

Description

### Connector Name

source-klaviyo

### Connector Version

2.14.11

### What step the error happened?

During the sync

### Relevant information

Request & Bug: Configurable Lookback Window for Incremental Syncs (to Handle Delayed Attribution in Klaviyo Events)


Context:

Klaviyo events such as purchases often have attribution (e.g., campaign source). Currently, this data is added to the event instantly, and for some events there is an delay in adding this data. However, in the near future, this will have a delay of 3 hours for all events with attribution data. Klaviyo events are immutable and do not have an updated_at field. The main timestamp available (datetime) reflects when the event occurred, not when attribution or other enrichment is added.


This causes Airbyte’s incremental syncs—which use datetime as the cursor—to miss important attribution data that arrives after the initial event is extracted. See Klaviyo docs:



The problem:




  • When events are enriched after creation (e.g., attribution added 3h after purchase), their datetime does not change.




  • Incremental syncs based on datetime will not re-sync these updated events.




  • There is no updated_at or similar field to use for an accurate cursor.




Proposed Solution: Configurable Lookback Window




  • Add a user-configurable “lookback window” to the Klaviyo source connector. Preferably, only for specific user defined events.




  • On every incremental sync, re-fetch all events where datetime is within N hours before the last synced timestamp.




    • Example: If last sync fetched up to 2024-05-30T10:00:00Z, next sync would fetch from 2024-05-30T04:00:00Z onward (with a 6h window).






  • UI/Config:




    • Expose the window as a setting in the Airbyte connector config, with a safe default (e.g., 6 hours).




    • Update documentation to warn users about delayed attribution and recommend enabling this setting for full attribution data.






Benefits:




  • Prevents data loss due to late-arriving or enriched events.




  • Flexible—users can adjust the window based on observed data patterns.



### Relevant log output

```shell

```

### Contribute

- [ ] Yes, I want to contribute

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.