Azure / Azure/Azure-DataFactory

[Question] [Copy Activity with Avro file sink] Is it possible to get incomplete state in sink location in case of source failure?

Open
#688 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
PowerShell
Stars
529
Forks
623
PR merge metrics
No merged PRs in 30d

Description

In our pipeline we want to use Copy Activity to move data from SAP HANA source into Avro file sink pointing to a unique location in ADLSv2.
Activity is statically configured with MaxRowsPerFile = 500000 and may be dynamically configured for parallel copy.
We anticipate that source might sometimes fail due to transient connectivity issues and want to configure the activity to perform a number of retries.
At this point we want to understand: is it possible to get incomplete state in sink location in case of source failure? And if its possible, what are our options to keep sink location empty between retries, or before next pipeline run, if all retry attempts ended with failure?

Thanks for the help!
-s

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.