Azure / Azure/Azure-DataFactory

Improper Deletes in Delta Format

Open
#560 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
PowerShell
Stars
529
Forks
623
PR merge metrics
No merged PRs in 30d

Description

I have a simple Data Flow that is writing to an inline Delta sink that writes to an ADLS Gen 2 storage account. Before I output to the sink, I use an Alter Row modifier to ensure each row is properly inserted into, updated in, or removed from the table. Unfortunately, when I ran the Data Flow as the sole Pipeline Activity, it writes one parquet file with all of the rows and records it as an `add` action in the transaction log. I would have expected instead to see the rows properly dispersed into multiple parquet files with the appropriate `add` and `remove` action.

In my small example, I was writing into an empty directory and the rows consist of the same key being added and removed multiple times. Am I missing something in my pipeline?

image

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.