Azure / Azure/Azure-DataFactory

Data Flow with ADLS source has missed some partitions

Open
#586 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
PowerShell
Stars
529
Forks
623
PR merge metrics
No merged PRs in 30d

Description

We use Data Factory to synchronise data from a Synapse Link Data Lake with an Azure SQL database. Each time the process runs, a mapping Data Flow is called for each table. This filters rows using the maximum _SinkModifiedOn_ from the previous run. Database users have informed us that some rows on one of the tables are out of date. I have traced this back to a particular run of the Data Flow. I then ran the Data Flow in debug mode to try and simulate what it did:

In debug mode, the data preview returns 172 rows. However, when it ran for real, it picked up only 139 rows. Drilling down into the run statistics I can see that partitions 2017 to 2023 account for these 139 rows. So it appears that partitions 2009 to 2016 have not been picked up although there are changes in these partitions.

Any idea why this would be? I cannot see this error on any other Data Flows for this table (or any others).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.