airbytehq / airbytehq/airbyte

Bug - Source connector made with connector Builder do not recognize the last date modification cursor in the connection state "streamState" in incrementa sync

Abierto
#61,567 6 comentarios 0 reacciones 1 asignado Reclamado por @lazebnyi Ver en GitHub
area/connector-builder community team/use
Lenguaje dominante
Python
Estrellas
22.1k
Forks
5.3k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

### Topic

Connector Builder bug in incremental sync conections

### Relevant information

Hello community.

I have developed a source connector for Dolibarr ERP in the Connector Builder UI, in this connector I configured several parent streams that support standar pagination and incremental sync for an API with No Filter by date capability, also I configured some child streams that depends on the parent streams and in this child streams I enable the switch indicating that their paren stream have incremental sync and validated that the parent stream cursor is updated when the child stream data is modified or updated.
For the incremental sync in the parent streams I configured the `sortfield` with the corresponding `date_modification` parameter and `sortorder` in descendent as required by Airbyte for incremental sync without date filtering. Also I configured the cursor field for the incremental sync with the corresponding date_modification variable for each parent stream.
The problem I identified is that in the first sync with this connector with the `Incremental | Append + Dedup` enabled in the parent streams the sincronization works well and the `streamState` for the connection is saved correctly with the last date modification of the data extracted but, when I make some changes in the origin data and try the second sync, the connector again extract all the records from the child streams and the first page of the parent streams and take so long time to do the extract, load and deduplication (much more than in the first sync) as if the connector where ignoring the last `streamState`. The AI agent in the slack community guided me in all verifications and the conclusion is a possible bug in the Connector Builder that generates a logic for the conector that ignores the last `streamState` for the incrementas sync in incremental parent streams with child streams.

I attach the YAML code in a txt file of the connector and the link to the PR of the Dolibarr connector i have developed:
https://github.com/airbytehq/airbyte/pull/61388

[dolibarr.txt](https://github.com/user-attachments/files/20720937/dolibarr.txt)

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.