Azure / Azure/Azure-DataFactory
Improper Deletes in Delta Format
- Dominant language
- PowerShell
- Stars
- 529
- Forks
- 623
- PR merge metrics
- No merged PRs in 30d
Description
I have a simple Data Flow that is writing to an inline Delta sink that writes to an ADLS Gen 2 storage account. Before I output to the sink, I use an Alter Row modifier to ensure each row is properly inserted into, updated in, or removed from the table. Unfortunately, when I ran the Data Flow as the sole Pipeline Activity, it writes one parquet file with all of the rows and records it as an `add` action in the transaction log. I would have expected instead to see the rows properly dispersed into multiple parquet files with the appropriate `add` and `remove` action.
In my small example, I was writing into an empty directory and the rows consist of the same key being added and removed multiple times. Am I missing something in my pipeline?
Contributor guide
Assessment
This issue has not been assessed yet.