Azure / Azure/Azure-DataFactory
Exclude Managed Private Endpoints from ARM Template Export
- Dominant language
- PowerShell
- Stars
- 529
- Forks
- 623
- PR merge metrics
- No merged PRs in 30d
Description
### Problem
The ARM Template exported from Data Factory using the Git integration includes the Managed Private Endpoint resources. As these Private Endpoints are environment specific, these should not be promoted to other environments and should be excluded from the ARM template. In the current state, all MPEs are propagated up to UAT and Production, meaning that the UAT and Production Data Factory instances try to create Managed Private Endpoints to the DEV SQL and Cosmos DB Account.
Our Terraform instances are provisioned via Terraform, and Managed Private Endpoints are configured there, with DEV connections to DEV resources (SQL Servers, Storage Accounts, Key Vaults, Cosmos DB etc.), UAT to UAT resources and Production to Production resources.
**A stripped-down version of the ARM Template**
````json
{
"name": "[concat(parameters('factoryName'), '/default')]",
"type": "Microsoft.DataFactory/factories/managedVirtualNetworks",
"apiVersion": "2018-06-01",
"properties": {},
"dependsOn": []
},
{
"name": "[concat(parameters('factoryName'), '/default/cosmosdb-dev01-server01-CosmosDb-Sql')]",
"type": "Microsoft.DataFactory/factories/managedVirtualNetworks/managedPrivateEndpoints",
"apiVersion": "2018-06-01",
"properties": {
"privateLinkResourceId": "[parameters('cosmosdb-dev01-server01-CosmosDb-Sql_properties_privateLinkResourceId')]",
"groupId": "[parameters('cosmosdb-dev01-server01-CosmosDb-Sql_properties_groupId')]",
"fqdns": "[parameters('cosmosdb-dev01-server01-CosmosDb-Sql_properties_fqdns')]"
},
"dependsOn": [
"[concat(variables('factoryId'), '/managedVirtualNetworks/default')]"
]
},
{
"name": "[concat(parameters('factoryName'), '/default/sql-dev01-server01-SqlServer-sqlServer')]",
"type": "Microsoft.DataFactory/factories/managedVirtualNetworks/managedPrivateEndpoints",
"apiVersion": "2018-06-01",
"properties": {
"privateLinkResourceId": "[parameters('sql-dev01-server01-SqlServer-sqlServer_properties_privateLinkResourceId')]",
"groupId": "[parameters('sql-dev01-server01-SqlServer-sqlServer_properties_groupId')]",
"fqdns": "[parameters('sql-dev01-server01-SqlServer-sqlServer_properties_fqdns')]"
},
"dependsOn": [
"[concat(variables('factoryId'), '/managedVirtualNetworks/default')]"
]
}
````
We are using this document [Sample pre- and post-deployment script](https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-sample-script) to promote changes from development through to production. The workflow looks like this:
- Data Factory deployed via Terraform.
- Managed Private Endpoints created and managed via Terraform with DEV Data Factory connected to DEV resource (databases, storage accounts etc.)
- Changes published to Git repo using Git integration / vsts_configuration. Repo has the following folders:
````
DF-DEV01-MYDATAFACTORY01
├── globalParameters/
│ └── DF-DEV01-MYDATAFACTORY01_GlobalParameters.json
├── linkedTemplates/
│ ├── ArmTemplate_0.json
│ ├── ArmTemplate_master.json
│ └── ArmTemplateParameters_master.json
├── ARMTemplateForFactory.json
└── ARMTemplateParametersForFactory.json
azure-pipelines.yaml
````
- Pipeline with `PrePostDeploymentScript.ps1` triggered when ADF publishes to the Git repo to promote changes through to production.
### Solution
- Option 1: Data Factory to not export resources of type `Microsoft.DataFactory/factories/managedVirtualNetworks/managedPrivateEndpoints` to the ARM Template.
- Option 2: `PrePostDeploymentScript.Ver2.ps1` script to have an option to strip out `Microsoft.DataFactory/factories/managedVirtualNetworks/managedPrivateEndpoints` resources.
Contributor guide
Assessment
This issue has not been assessed yet.