[source-salesforce] Incremental stream loosing data on schema change
- Vorherrschende Sprache
- Python
- Sterne
- 22.1k
- Forks
- 5.4k
- Ø Merge
- 5 Std.
- Gemergte PRs (30 T.)
- 671
Beschreibung
### Connector Name
source-salesforce
### Connector Version
2.7.16
### What step the error happened?
During the sync
### Relevant information
I am using Airbyte OS. Among other custom and marketplace sources, I am using the Salesforce connector.
I setup several streams to be synced (Incremental + Deduped) into Snowflake, and it is generally working fine.
Recently, I experienced a data loss on one of these streams (the Opportunity entities stream).
In particular:
- the row count in the destination table in Snowflake is lower than expected
- it is lower than the record count on Salesforce at the source, and lower than what I remember I had in the destination table before the problem (also proven by data quality monitors that detected a sudden count decrease)
- Looking into the corresponding `airbyte_internal` table, I also observed a decrease of records. Which is surprising, because I thought this table is append-only
Another peculiar thing that happened is about the distribution of the creation date of the records.
Looking at the table with missing records with this query
```
select
count(*),
date_trunc('month', createddate)
from raw.salesforce.opportunity
group by 2;
```
it seemed like the vast majority of the Opportunities where created in August 2025.
Which I know to be simply not true.
A sanity check in the other table (the one I re-synced from scratch) showed a distribution that
corresponds more to reality
(I am sorry I can't share exact numbers here; but the two graphs should give you and idea of the problem)
I tried re-syncing from scratch using a separate connection (therefore, a separate destination table), and I could see that the sync is able to fetch all the records I expect (the corresponding `airbyte_internal` table is much larger, and the row count in the destination table corresponds to what I have at the source).
The only special thing that happened to this stream in concomitance with this problem was a schema change.
The Salesforce team added a new column, and the connection settings are **"Propagate field changes only"**.
To mitigate the problem, given what I observed and my experiment with a separate connection, I think I can just refresh this individual stream and then proceed with the incremental syncs, but I would like to understand the root cause of the problem:
- how can data be deleted
- how the `creationdate` can be mangled upon a sync
Unfortunately the episode is already outside of the window of time for which I have Snowflake logs, so I cannot provide the queries that were executed by Airbyte.
Thank you for any clarifications you can provide, and happy to follow up with more details if needed.
### Relevant log output
```shell
```
### Contribute
- [ ] Yes, I want to contribute
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/12132
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.