Azure / Azure/Azure-DataFactory
In Azure Data Factory, can I pass data from dataset as input to the Azure Function?
- Dominant language
- PowerShell
- Stars
- 529
- Forks
- 623
- PR merge metrics
- No merged PRs in 30d
Description
I am able to copy data between two 'Azure Cosmos DB for MongoDB' datasets using Azure Data Factory pipeline.
Now, I would like to transform the data using Azure Function before the data is pushed to the destination dataset. Is this possible in Azure Data Factory? If so, is there a documentation that I can refer to?
I thought this should have been a supported use case. Each record from the source dataset, considering it is in JSON format could be forwarded as-is in the request body to the HTTP triggered function, and the JSON response of this function would be available for use later in the pipeline.
I tried setting the body of the HTTP triggered function as the @activity('Copy Activity').output but the 'Copy Activity' output contains the metadata about the activity execution, not the dataset records that I am looking for. I also gave a try to dataflows but could not get started as 'Azure Cosmos DB for Mongo DB' is [not a supported data store](https://learn.microsoft.com/en-us/azure/data-factory/connector-overview#supported-data-stores). There is not much information that can help with my scenario in the [Azure Function activity](https://learn.microsoft.com/en-us/azure/data-factory/control-flow-azure-function-activity) documentation page.
Contributor guide
Assessment
This issue has not been assessed yet.