airbytehq / airbytehq/airbyte

[source-salesforce] Incremental stream loosing data on schema change

Offen
#77,627 6 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
area/connectors autoteam community connectors/source/salesforce needs-triage team/use type/bug
Vorherrschende Sprache
Python
Sterne
22.1k
Forks
5.4k
Ø Merge
5 Std.
Gemergte PRs (30 T.)
671

Beschreibung

### Connector Name

source-salesforce

### Connector Version

2.7.16

### What step the error happened?

During the sync

### Relevant information

I am using Airbyte OS. Among other custom and marketplace sources, I am using the Salesforce connector.
I setup several streams to be synced (Incremental + Deduped) into Snowflake, and it is generally working fine.

Recently, I experienced a data loss on one of these streams (the Opportunity entities stream).
In particular:

- the row count in the destination table in Snowflake is lower than expected
- it is lower than the record count on Salesforce at the source, and lower than what I remember I had in the destination table before the problem (also proven by data quality monitors that detected a sudden count decrease)
- Looking into the corresponding `airbyte_internal` table, I also observed a decrease of records. Which is surprising, because I thought this table is append-only

Another peculiar thing that happened is about the distribution of the creation date of the records.
Looking at the table with missing records with this query

```
select
count(*),
date_trunc('month', createddate)
from raw.salesforce.opportunity
group by 2;
```

it seemed like the vast majority of the Opportunities where created in August 2025.
Which I know to be simply not true.

Image

A sanity check in the other table (the one I re-synced from scratch) showed a distribution that
corresponds more to reality

Image

(I am sorry I can't share exact numbers here; but the two graphs should give you and idea of the problem)

I tried re-syncing from scratch using a separate connection (therefore, a separate destination table), and I could see that the sync is able to fetch all the records I expect (the corresponding `airbyte_internal` table is much larger, and the row count in the destination table corresponds to what I have at the source).

The only special thing that happened to this stream in concomitance with this problem was a schema change.
The Salesforce team added a new column, and the connection settings are **"Propagate field changes only"**.

To mitigate the problem, given what I observed and my experiment with a separate connection, I think I can just refresh this individual stream and then proceed with the incremental syncs, but I would like to understand the root cause of the problem:

- how can data be deleted
- how the `creationdate` can be mangled upon a sync

Unfortunately the episode is already outside of the window of time for which I have Snowflake logs, so I cannot provide the queries that were executed by Airbyte.

Thank you for any clarifications you can provide, and happy to follow up with more details if needed.

### Relevant log output

```shell

```

### Contribute

- [ ] Yes, I want to contribute

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/12132

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.