Azure / Azure/Azure-DataFactory

Source/Sink behaving differently when using parameterized Datasets

Open
#656 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
PowerShell
Stars
529
Forks
623
PR merge metrics
No merged PRs in 30d

Description

I use parameters with datasets in Azure Data Factory. What I've noticed is that parameters behave differently in a dataflow sink versus a dataflow source. For example, this is the parameters I set at the pipeline level on a dataflow:

![image](https://github.com/Azure/Azure-DataFactory/assets/2133717/d5d740e8-d69a-4d23-afea-c55005ea4ef4)

I provide dynamically generated filenames to both the source and sink. However, while the source is able to read in the filename from the storage blob just fine, the sink does not and I have to "override" it as such:

![image](https://github.com/Azure/Azure-DataFactory/assets/2133717/f678c81c-e296-483d-a987-f526584c152f)

If I don't, the sink writes the file in the RIGHT folder, mind you, but with the filename of `part-` and subsequent steps in the pipeline fail because they can't find the file.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.