Azure / Azure/azure-powershell

New-AzDataCollectionRule. does not create DCR endoints (logs& Metrics)

Open
#25,727 14 comments 0 reactions 0 assignees View on GitHub
act-observability-squad Azure PS Team customer-reported feature-request Monitor Tracking
Dominant language
C#
Stars
4.8k
Forks
4.3k
Avg merge
3d 14h
Merged PRs (30d)
54

Description

### Description

Hello
Contrary to ARM deployment the DCR logs and metrics endpoints are not generated when created a new DRC with PowerShell

### Issue script & Debug output

```PowerShell
New-AzDataCollectionRule -ResourceGroupName 'RGxxxxx' -Name 'DCR-ReproTest' -JsonFilePath '/home/azadm/New_DCR_AZPolicyComplianceDetails.json'
DEBUG: 11:45:31 AM - [ConfigManager] Got [True] from [DisplaySecretsWarning], Module = [], Cmdlet = [].
DEBUG: 11:45:31 AM - GetAzureRMContextCommand begin processing with ParameterSet 'GetSingleContext'.
DEBUG: 11:45:31 AM - [ConfigManager] Got [False] from [DisplayBreakingChangeWarning], Module = [], Cmdlet = [].
DEBUG: 11:45:31 AM - [ConfigManager] Got [True] from [DisplaySecretsWarning], Module = [], Cmdlet = [].
DEBUG: 11:45:31 AM - [ConfigManager] Got nothing from [DisplayRegionIdentified], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: 11:45:31 AM - [ConfigManager] Got nothing from [CheckForUpgrade], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: 11:45:31 AM - GetAzureRMContextCommand end processing.
DEBUG: [CmdletBeginProcessing]: Starting command
DEBUG: CmdletBeginProcessing:
DEBUG: CmdletProcessRecordStart:
DEBUG: CmdletGetPipeline:
DEBUG: CmdletBeforeAPICall:
DEBUG: URLCreated: /subscriptions/yyyyyy/resourceGroups/RGxxxxxx/providers/Microsoft.Insights/dataCollectionRules/DCR-ReproTest?api-version=2022-06-01
DEBUG: RequestCreated: /subscriptions/yyyyyy/resourceGroups/RGxxxxxx/providers/Microsoft.Insights/dataCollectionRules/DCR-ReproTest?api-version=2022-06-01
DEBUG: HeaderParametersAdded:
DEBUG: BodyContentSet:
DEBUG: 11:45:31 AM - [ConfigManager] Got nothing from [DisableInstanceDiscovery], Module = [], Cmdlet = []. Returning default value [False].
DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
PUT

Absolute Uri:
https://management.azure.com/subscriptions/yyyy/resourceGroups/RGxxxxxxx/providers/Microsoft.Insights/dataCollectionRules/DCR-ReproTest?api-version=2022-06-01

Headers:
x-ms-unique-id : 2
x-ms-client-request-id :yyyyyyyy
CommandName : New-AzDataCollectionRule
FullCommandName : New-AzDataCollectionRule_CreateViaJsonFilePath
ParameterSetName : __AllParameterSets
User-Agent : AzurePowershell/v12.1.0,PSVersion/v7.4.3,Az.DataCollectionRule/5.2.1

Body:
{
"location": "westeurope",
"properties": {
"streamDeclarations": {
"Custom-Historical_AzPolicyComplianceDetails_CL": {
"columns": [
{
"name": "policyAssignmentId",
"type": "string"
},
{
"name": "policyDefinitionId",
"type": "string"
},
{
"name": "policyDefinitionReferenceId",
"type": "string"
},
{
"name": "policyDefinitionGroupNames",
"type": "string"
},
{
"name": "policyDefinitionAction",
"type": "string"
},
{
"name": "numberOfNonCompliantResources",
"type": "int"
},
{
"name": "numberOfCompliantResources",
"type": "int"
},
{
"name": "details",
"type": "dynamic"
}
]
}
},
"destinations": {
"logAnalytics": [
{
"workspaceResourceId": "/subscriptions/xxxxxx/resourceGroups/rgxxxxxxx/providers/microsoft.operationalinsights/workspaces/policyworkspace",
"workspaceId": "xxxx",
"name": "myworkspace"
}
]
},
"dataFlows": [
{
"streams": [
"Custom-Historical_AzPolicyComplianceDetails_CL"
],
"destinations": [
"myworkspace"
],
"transformKql": "source\n| extend TimeGenerated = now()\n",
"outputStream": "Custom-Historical_AzPolicyComplianceDetails_CL"
}
]
}
}

DEBUG: BeforeCall:
DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Cache-Control : no-cache
Pragma : no-cache
Vary : Accept-Encoding
x-ms-ratelimit-remaining-subscription-resource-requests: 149
Request-Context : appId=cid-v1:x
x-ms-correlation-request-id : x
x-ms-client-request-id : x
x-ms-routing-request-id : WESTEUROPE:xx
x-ms-request-id : xxxx
api-supported-versions : 2019-11-01-preview, 2021-04-01, 2021-09-01-preview, 2022-06-01, 2023-03-11, 2024-03-11
Strict-Transport-Security : max-age=31536000; includeSubDomains
X-Content-Type-Options : nosniff
X-Cache : CONFIG_NOCACHE
X-MSEdge-Ref : Ref A: xxxx Ref B: xxxx Ref C: 2024-08-01T11:45:31Z
Date : Thu, 01 Aug 2024 11:45:33 GMT

Body:
{
"properties": {
"immutableId": "dcr-ewfwff3",
"streamDeclarations": {
"Custom-Historical_AzPolicyComplianceDetails_CL": {
"columns": [
{
"name": "policyAssignmentId",
"type": "string"
},
{
"name": "policyDefinitionId",
"type": "string"
},
{
"name": "policyDefinitionReferenceId",
"type": "string"
},
{
"name": "policyDefinitionGroupNames",
"type": "string"
},
{
"name": "policyDefinitionAction",
"type": "string"
},
{
"name": "numberOfNonCompliantResources",
"type": "int"
},
{
"name": "numberOfCompliantResources",
"type": "int"
},
{
"name": "details",
"type": "dynamic"
}
]
}
},
"destinations": {
"logAnalytics": [
{
"workspaceResourceId": "/subscriptions/9yyyyy/resourceGroups/rg-int-dgs-lab-its-itinfra-1/providers/microsoft.operationalinsights/workspaces/policyworkspace",
"workspaceId": "zzzzzz",
"name": "myworkspace"
}
]
},
"dataFlows": [
{
"streams": [
"Custom-Historical_AzPolicyComplianceDetails_CL"
],
"destinations": [
"myworkspace"
],
"transformKql": "source\n| extend TimeGenerated = now()\n",
"outputStream": "Custom-Historical_AzPolicyComplianceDetails_CL"
}
],
"provisioningState": "Succeeded"
},
"location": "westeurope",
"id": "/subscriptions/yyyyy/resourceGroups/RGxxxxx/providers/Microsoft.Insights/dataCollectionRules/DCR-ReproTest",
"name": "DCR-ReproTest",
"type": "Microsoft.Insights/dataCollectionRules",
"etag": "\"b6009126-0000-0d00-0000-66ab755d0000\"",
"systemData": {
"createdBy": "xxxxx",
"createdByType": "User",
"createdAt": "2024-08-01T11:45:31.8835297Z",
"lastModifiedBy": "xxxxx",
"lastModifiedByType": "User",
"lastModifiedAt": "2024-08-01T11:45:31.8835297Z"
}
}

DEBUG: ResponseCreated:
DEBUG: BeforeResponseDispatch:
DEBUG: Finally:
DEBUG: CmdletAfterAPICall:
DEBUG: [CmdletProcessRecordAsyncEnd]: Finish HTTP process
DEBUG: CmdletProcessRecordAsyncEnd:
DEBUG: CmdletProcessRecordEnd:
DEBUG: 11:45:33 AM - [ConfigManager] Got [True] from [DisplaySecretsWarning], Module = [], Cmdlet = [].

DataCollectionEndpointId :
DataFlow : {{
"streams": [ "Custom-Historical_AzPolicyComplianceDetails_CL" ],
"destinations": [ "myworkspace" ],
"transformKql": "source\n| extend TimeGenerated = now()\n",
"outputStream": "Custom-Historical_AzPolicyComplianceDetails_CL"
}}
DataSourceDataImportEventHubConsumerGroup :
DataSourceDataImportEventHubName :
DataSourceDataImportEventHubStream :
DataSourceExtension :
DataSourceIisLog :
DataSourceLogFile :
DataSourcePerformanceCounter :
DataSourcePlatformTelemetry :
DataSourcePrometheusForwarder :
DataSourceSyslog :
DataSourceWindowsEventLog :
DataSourceWindowsFirewallLog :
Description :
DestinationAzureMonitorMetricName :
DestinationEventHub :
DestinationEventHubsDirect :
DestinationLogAnalytic : {{
"workspaceResourceId": "/subscriptions/xxxxxxx/resourceGroups/rgxxxxxx/providers/microsoft.operationalinsights/workspaces/policyworkspace",
"workspaceId": "4yyyyy",
"name": "myworkspace"
}}
DestinationMonitoringAccount :
DestinationStorageAccount :
DestinationStorageBlobsDirect :
DestinationStorageTablesDirect :
Etag : "b6009126-0000-0d00-0000-66ab755d0000"
Id : /subscriptions/9XXXXX/resourceGroups/RGXXXXXX1/providers/Microsoft.Insights/dataCollectionRules/DCR-ReproTest
IdentityPrincipalId :
IdentityTenantId :
IdentityType :
IdentityUserAssignedIdentity : {
}
ImmutableId : dcr-2e40a7469fXXXXXX
Kind :
Location : westeurope
MetadataProvisionedBy :
MetadataProvisionedByResourceId :
Name : DCR-ReproTest
ProvisioningState : Succeeded
ResourceGroupName : RGXXXX
StreamDeclaration : {
"Custom-Historical_AzPolicyComplianceDetails_CL": {
"columns": [
{
"name": "policyAssignmentId",
"type": "string"
},
{
"name": "policyDefinitionId",
"type": "string"
},
{
"name": "policyDefinitionReferenceId",
"type": "string"
},
{
"name": "policyDefinitionGroupNames",
"type": "string"
},
{
"name": "policyDefinitionAction",
"type": "string"
},
{
"name": "numberOfNonCompliantResources",
"type": "int"
},
{
"name": "numberOfCompliantResources",
"type": "int"
},
{
"name": "details",
"type": "dynamic"
}
]
}
}
SystemDataCreatedAt : 8/1/2024 11:45:31 AM
SystemDataCreatedBy : xxxx
SystemDataCreatedByType : User
SystemDataLastModifiedAt : 8/1/2024 11:45:31 AM
SystemDataLastModifiedBy : xxx
SystemDataLastModifiedByType : User
Tag : {
}
Type : Microsoft.Insights/dataCollectionRules

DEBUG: AzureQoSEvent: Module: Az.Monitor:5.2.1; CommandName: New-AzDataCollectionRule; PSVersion: 7.4.3; IsSuccess: True; Duration: 00:00:02.5406602; SanitizeDuration: 00:00:00.0186410
```

### Environment data

```PowerShell
Name Value
---- -----
PSVersion 7.4.3
PSEdition Core
GitCommitId 7.4.3
OS CBL-Mariner/Linux
Platform Unix
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0
```

### Module versions

```PowerShell
ModuleType Version PreRelease Name ExportedCommands
---------- ------- ---------- ---- ----------------
Script 3.0.2 Az.Accounts {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-AzDefault…}
Script 8.0.0 Az.Compute {Add-AzImageDataDisk, Add-AzVhd, Add-AzVMAdditionalUnattendContent, Add-AzVMDataDisk…}
Script 7.8.0 Az.Network {Add-AzApplicationGatewayAuthenticationCertificate, Add-AzApplicationGatewayBackendAddressPool, Add-AzApplicationGatewayBackendHttpSetting, Add-AzApplicationGatewayBackendSetting…}
Script 7.1.0 Az.Resources {Export-AzResourceGroup, Export-AzTemplateSpec, Get-AzDenyAssignment, Get-AzDeployment…}
Script 7.0.0 Az.Storage {Add-AzRmStorageContainerLegalHold, Add-AzStorageAccountManagementPolicyAction, Add-AzStorageAccountNetworkRule, Close-AzStorageFileHandle…}
Script 1.1.3 Az.Tools.Predictor {Disable-AzPredictor, Enable-AzPredictor, Open-AzPredictorSurvey, Send-AzPredictorRating}
Script 0.0.0.10 AzureAD.Standard.Preview {Add-AzureADApplicationOwner, Add-AzureADDeviceRegisteredOwner, Add-AzureADDeviceRegisteredUser, Add-AzureADDirectoryRoleMember…}
Script 0.9.3 AzurePSDrive
```

### Error output

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with New-AzDataCollectionRule, the supplied JSON file, and the logged 2022-06-01 request, then compare the result with an ARM deployment. Trace how the cmdlet represents the returned DCR and its logs and metrics endpoints; done means the PowerShell-created DCR contains the expected endpoints and the behavior is covered by an appropriate regression check.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, powershell
Domain
cli, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.