Azure / Azure/Azure-DataFactory

In Azure Data Factory, can I pass data from dataset as input to the Azure Function?

Open
#614 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
PowerShell
Stars
529
Forks
623
PR merge metrics
No merged PRs in 30d

Description

I am able to copy data between two 'Azure Cosmos DB for MongoDB' datasets using Azure Data Factory pipeline.

Now, I would like to transform the data using Azure Function before the data is pushed to the destination dataset. Is this possible in Azure Data Factory? If so, is there a documentation that I can refer to?

I thought this should have been a supported use case. Each record from the source dataset, considering it is in JSON format could be forwarded as-is in the request body to the HTTP triggered function, and the JSON response of this function would be available for use later in the pipeline.

I tried setting the body of the HTTP triggered function as the @activity('Copy Activity').output but the 'Copy Activity' output contains the metadata about the activity execution, not the dataset records that I am looking for. I also gave a try to dataflows but could not get started as 'Azure Cosmos DB for Mongo DB' is [not a supported data store](https://learn.microsoft.com/en-us/azure/data-factory/connector-overview#supported-data-stores). There is not much information that can help with my scenario in the [Azure Function activity](https://learn.microsoft.com/en-us/azure/data-factory/control-flow-azure-function-activity) documentation page.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.