Azure / Azure/azure-powershell
Az.Monitor LogIngestion in Custom Tables: Output stream not generated
- Dominant language
- C#
- Stars
- 4.8k
- Forks
- 4.3k
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 51
Description
### Description
I create Custom Tables in my Log Analytics Workspace. I also create a Data Collection Endpoint and Data Collection Rule.
When I create my data collection rule, it is successfully created and so are the Data Sources. However the Destination of each Data Source is empty.
I tried to go by this tutorial: https://learn.microsoft.com/en-us/azure/azure-monitor/logs/tutorial-logs-ingestion-api
Exporting the template in the Azure Portal lacks the transformKql and the outputstream.
Data Sources Tab of Data Collection Rule in Azure Portal:

### Issue script & Debug output
```PowerShell
My Custom Table look like this:
`{
"properties": {
"schema": {
"name": "Karl3_CL",
"columns": [
{
"name": "TimeGenerated",
"type": "datetime"
},
{
"name": "EventType",
"type": "string"
},
{
"name": "ApplicationName",
"type": "string"
},
{
"name": "EventId",
"type": "string"
},
{
"name": "ApplicationId",
"type": "string"
},
{
"name": "EventData",
"type": "string"
}
]
}
}
}`
Template (properties section) used for creation:
`{
"properties": {
"dataCollectionEndpointId": "redacted",
"streamDeclarations": {
"Custom-Karl3_CL": {
"columns": [
{
"name": "TimeGenerated",
"type": "datetime"
},
{
"name": "EventType",
"type": "string"
},
{
"name": "ApplicationName",
"type": "string"
},
{
"name": "EventId",
"type": "string"
},
{
"name": "ApplicationId",
"type": "string"
},
{
"name": "EventData",
"type": "string"
}
]
}
},
"dataSources": {},
"destinations": {
"logAnalytics": [
{
"workspaceResourceId": "redacted",
"name": "b4414f66b1fc48e7b33a2c9ead545134"
}
]
},
"dataFlows": [
{
"destinations": [
"b4414f66b1fc48e7b33a2c9ead545134"
],
"streams": [
"Custom-Karl3_CL"
],
"transformKql": "source",
"outputStream": "Custom-Karl3_CL"
}
]
}
}
`
Template after creation (from Azure Portal):
`{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"dataCollectionRules_resourcename_name": {
"defaultValue": "redacted",
"type": "String"
},
"workspaces_resourcename_externalid": {
"defaultValue": "redacted",
"type": "String"
}
},
"variables": {},
"resources": [
{
"type": "Microsoft.Insights/dataCollectionRules",
"apiVersion": "2022-06-01",
"name": "[parameters('dataCollectionRules_resourcename_name')]",
"location": "West Europe",
"tags": {
"Owner": "Karl"
},
"properties": {
"dataSources": {},
"destinations": {
"logAnalytics": [
{
"workspaceResourceId": "[parameters('workspaces_resourcename_externalid')]",
"name": "b4414f66b1fc48e7b33a2c9ead545134"
}
]
},
"dataFlows": [
{
"streams": [
"Custom-Karl3_CL"
],
"destinations": [
"b4414f66b1fc48e7b33a2c9ead545134"
]
}
]
}
}
]
}`
I create the datacollection rule using this function:
$dataCollectionRule = New-AzDataCollectionRule `
-ResourceGroupName $ResourceGroupName `
-RuleName $DataCollectionRuleName `
-Location $Location `
-RuleFile $dcrRuleFilePath
```
### Environment data
```PowerShell
Name Value
---- -----
PSVersion 7.3.4
PSEdition Core
GitCommitId 7.3.4
OS Microsoft Windows 10.0.19044
Platform Win32NT
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0
```
### Module versions
```PowerShell
ModuleType Version PreRelease Name ExportedCommands
---------- ------- ---------- ---- ----------------
Script 9.1.1 Az
Script 2.10.4 Az.Accounts {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-AzDefault…}
Script 2.0.0 Az.Advisor {Disable-AzAdvisorRecommendation, Enable-AzAdvisorRecommendation, Get-AzAdvisorConfiguration, Get-AzAdvisorRecommendation…}
Script 5.0.1 Az.Aks {Disable-AzAksAddOn, Enable-AzAksAddOn, Get-AzAksCluster, Get-AzAksNodePool…}
Script 1.1.4 Az.AnalysisServices {Add-AzAnalysisServicesAccount, Export-AzAnalysisServicesInstanceLog, Get-AzAnalysisServicesServer, New-AzAnalysisServicesFirewallConfig…}
Script 4.0.1 Az.ApiManagement {Add-AzApiManagementApiToGateway, Add-AzApiManagementApiToProduct, Add-AzApiManagementProductToGroup, Add-AzApiManagementRegion…}
Script 1.2.0 Az.AppConfiguration {Clear-AzAppConfigurationDeletedStore, Get-AzAppConfigurationDeletedStore, Get-AzAppConfigurationStore, Get-AzAppConfigurationStoreKey…}
Script 2.1.0 Az.ApplicationInsights {Get-AzApplicationInsights, Get-AzApplicationInsightsApiKey, Get-AzApplicationInsightsContinuousExport, Get-AzApplicationInsightsLinkedStorageAccount…}
Script 2.0.0 Az.Attestation {Add-AzAttestationPolicySigner, Get-AzAttestationPolicy, Get-AzAttestationPolicySigners, Remove-AzAttestationPolicySigner…}
Script 1.8.0 Az.Automation {Export-AzAutomationDscConfiguration, Export-AzAutomationDscNodeReportContent, Export-AzAutomationRunbook, Get-AzAutomationAccount…}
Script 3.2.1 Az.Batch {Disable-AzBatchAutoScale, Disable-AzBatchComputeNodeScheduling, Disable-AzBatchJob, Disable-AzBatchJobSchedule…}
Script 2.0.0 Az.Billing {Get-AzBillingAccount, Get-AzBillingInvoice, Get-AzBillingPeriod, Get-AzBillingProfile…}
Script 2.1.0 Az.Cdn {Clear-AzCdnEndpointContent, Clear-AzFrontDoorCdnEndpointContent, Disable-AzCdnCustomDomainCustomHttps, Enable-AzCdnCustomDomainCustomHttps…}
Script 1.1.0 Az.CloudService {Get-AzCloudService, Get-AzCloudServiceInstanceView, Get-AzCloudServiceNetworkInterface, Get-AzCloudServiceOSFamily…}
Script 1.12.0 Az.CognitiveServices {Add-AzCognitiveServicesAccountNetworkRule, Get-AzCognitiveServicesAccount, Get-AzCognitiveServicesAccountCommitmentPlan, Get-AzCognitiveServicesAccountDeployment…}
Script 5.1.1 Az.Compute {Add-AzImageDataDisk, Add-AzVhd, Add-AzVMAdditionalUnattendContent, Add-AzVMDataDisk…}
Script 1.0.0 Az.ConfidentialLedger {Get-AzConfidentialLedger, New-AzConfidentialLedger, New-AzConfidentialLedgerAADBasedSecurityPrincipalObject, New-AzConfidentialLedgerCertBasedSecurityPrincipalObject…}
Script 3.1.0 Az.ContainerInstance {Add-AzContainerInstanceOutput, Get-AzContainerGroup, Get-AzContainerInstanceCachedImage, Get-AzContainerInstanceCapability…}
Script 3.0.0 Az.ContainerRegistry {Connect-AzContainerRegistry, Get-AzContainerRegistry, Get-AzContainerRegistryCredential, Get-AzContainerRegistryManifest…}
Script 1.8.2 Az.CosmosDB {Get-AzCosmosDBAccount, Get-AzCosmosDBAccountKey, Get-AzCosmosDBCassandraKeyspace, Get-AzCosmosDBCassandraKeyspaceThroughput…}
Script 1.1.0 Az.DataBoxEdge {Get-AzDataBoxEdgeBandwidthSchedule, Get-AzDataBoxEdgeDevice, Get-AzDataBoxEdgeJob, Get-AzDataBoxEdgeOrder…}
Script 1.4.0 Az.Databricks {Get-AzDatabricksAccessConnector, Get-AzDatabricksOutboundNetworkDependenciesEndpoint, Get-AzDatabricksVNetPeering, Get-AzDatabricksWorkspace…}
Script 1.16.10 Az.DataFactory {Add-AzDataFactoryV2DataFlowDebugSessionPackage, Add-AzDataFactoryV2TriggerSubscription, Get-AzDataFactory, Get-AzDataFactoryActivityWindow…}
Script 1.0.2 Az.DataLakeAnalytics {Add-AzDataLakeAnalyticsDataSource, Add-AzDataLakeAnalyticsFirewallRule, Get-AzDataLakeAnalyticsAccount, Get-AzDataLakeAnalyticsCatalogItem…}
Script 1.3.0 Az.DataLakeStore {Add-AzDataLakeStoreFirewallRule, Add-AzDataLakeStoreItemContent, Add-AzDataLakeStoreTrustedIdProvider, Add-AzDataLakeStoreVirtualNetworkRule…}
Script 1.0.0 Az.DataProtection {Backup-AzDataProtectionBackupInstanceAdhoc, Edit-AzDataProtectionPolicyRetentionRuleClientObject, Edit-AzDataProtectionPolicyTagClientObject, Edit-AzDataProtectionPolicyTriggerClientObject…}
Script 1.0.1 Az.DataShare {Get-AzDataShare, Get-AzDataShareAccount, Get-AzDataShareDataSet, Get-AzDataShareDataSetMapping…}
Script 1.1.0 Az.DeploymentManager {Get-AzDeploymentManagerArtifactSource, Get-AzDeploymentManagerRollout, Get-AzDeploymentManagerService, Get-AzDeploymentManagerServiceTopology…}
Script 3.1.1 Az.DesktopVirtualization {Disconnect-AzWvdUserSession, Expand-AzWvdMsixImage, Get-AzWvdApplication, Get-AzWvdApplicationGroup…}
Script 1.0.2 Az.DevTestLabs {Get-AzDtlAllowedVMSizesPolicy, Get-AzDtlAutoShutdownPolicy, Get-AzDtlAutoStartPolicy, Get-AzDtlVMsPerLabPolicy…}
Script 1.1.2 Az.Dns {Add-AzDnsRecordConfig, Get-AzDnsRecordSet, Get-AzDnsZone, New-AzDnsRecordConfig…}
Script 1.5.0 Az.EventGrid {Enable-AzEventGridPartnerTopic, Get-AzEventGridChannel, Get-AzEventGridDomain, Get-AzEventGridDomainKey…}
Script 3.1.0 Az.EventHub {Add-AzEventHubIPRule, Add-AzEventHubVirtualNetworkRule, Get-AzEventHubNamespace, New-AzEventHubAuthorizationRuleSASToken…}
Script 1.9.0 Az.FrontDoor {Disable-AzFrontDoorCustomDomainHttps, Enable-AzFrontDoorCustomDomainHttps, Get-AzFrontDoor, Get-AzFrontDoorFrontendEndpoint…}
Script 4.0.6 Az.Functions {Get-AzFunctionApp, Get-AzFunctionAppAvailableLocation, Get-AzFunctionAppPlan, Get-AzFunctionAppSetting…}
Script 5.0.1 Az.HDInsight {Add-AzHDInsightClusterIdentity, Add-AzHDInsightComponentVersion, Add-AzHDInsightConfigValue, Add-AzHDInsightMetastore…}
Script 2.0.0 Az.HealthcareApis {Get-AzHealthcareApisService, Get-AzHealthcareApisWorkspace, Get-AzHealthcareDicomService, Get-AzHealthcareFhirDestination…}
Script 2.7.4 Az.IotHub {Add-AzIotHubCertificate, Add-AzIotHubConfiguration, Add-AzIotHubDeployment, Add-AzIotHubDevice…}
Script 4.9.0 Az.KeyVault {Add-AzKeyVaultCertificate, Add-AzKeyVaultCertificateContact, Add-AzKeyVaultKey, Add-AzKeyVaultManagedStorageAccount…}
Script 2.1.0 Az.Kusto {Add-AzKustoClusterLanguageExtension, Add-AzKustoDatabasePrincipal, Get-AzKustoAttachedDatabaseConfiguration, Get-AzKustoCluster…}
Script 1.5.0 Az.LogicApp {Get-AzIntegrationAccount, Get-AzIntegrationAccountAgreement, Get-AzIntegrationAccountAssembly, Get-AzIntegrationAccountBatchConfiguration…}
Script 1.1.3 Az.MachineLearning {Add-AzMlWebServiceRegionalProperty, Export-AzMlWebService, Get-AzMlCommitmentAssociation, Get-AzMlCommitmentPlan…}
Script 1.2.1 Az.Maintenance {Get-AzApplyUpdate, Get-AzConfigurationAssignment, Get-AzMaintenanceConfiguration, Get-AzMaintenancePublicConfiguration…}
Script 1.1.0 Az.ManagedServiceIdentity {Get-AzFederatedIdentityCredentials, Get-AzSystemAssignedIdentity, Get-AzUserAssignedIdentity, Get-AzUserAssignedIdentityAssociatedResource…}
Script 3.0.0 Az.ManagedServices {Get-AzManagedServicesAssignment, Get-AzManagedServicesDefinition, Get-AzManagedServicesMarketplaceDefinition, New-AzManagedServicesAssignment…}
Script 2.0.0 Az.MarketplaceOrdering {Get-AzMarketplaceTerms, Invoke-AzMarketplaceSignTerms, Set-AzMarketplaceTerms, Stop-AzMarketplaceTerms}
Script 1.1.1 Az.Media {Get-AzMediaService, Get-AzMediaServiceKey, Get-AzMediaServiceNameAvailability, New-AzMediaService…}
Script 2.1.0 Az.Migrate {Get-AzMigrateDiscoveredServer, Get-AzMigrateJob, Get-AzMigrateProject, Get-AzMigrateReplicationFabric…}
Script 4.2.1 Az.Monitor {Add-AzLogProfile, Add-AzMetricAlertRule, Add-AzMetricAlertRuleV2, Add-AzWebtestAlertRule…}
Script 1.1.0 Az.MySql {Get-AzMySqlConfiguration, Get-AzMySqlConnectionString, Get-AzMySqlFirewallRule, Get-AzMySqlFlexibleServer…}
Script 5.1.2 Az.Network {Add-AzApplicationGatewayAuthenticationCertificate, Add-AzApplicationGatewayBackendAddressPool, Add-AzApplicationGatewayBackendHttpSetting, Add-AzApplicationGatewayBackendSetting…}
Script 1.1.1 Az.NotificationHubs {Get-AzNotificationHub, Get-AzNotificationHubAuthorizationRule, Get-AzNotificationHubListKey, Get-AzNotificationHubPNSCredential…}
Script 3.2.0 Az.OperationalInsights {Disable-AzOperationalInsightsIISLogCollection, Disable-AzOperationalInsightsLinuxCustomLogCollection, Disable-AzOperationalInsightsLinuxPerformanceCollection, Disable-AzOperationalInsightsLinuxSyslogCollection…}
Script 1.5.1 Az.PolicyInsights {Get-AzPolicyEvent, Get-AzPolicyMetadata, Get-AzPolicyRemediation, Get-AzPolicyState…}
Script 1.1.0 Az.PostgreSql {Get-AzPostgreSqlConfiguration, Get-AzPostgreSqlConnectionString, Get-AzPostgreSqlFirewallRule, Get-AzPostgreSqlFlexibleServer…}
Script 1.2.0 Az.PowerBIEmbedded {Get-AzPowerBIEmbeddedCapacity, Get-AzPowerBIWorkspace, Get-AzPowerBIWorkspaceCollection, Get-AzPowerBIWorkspaceCollectionAccessKey…}
Script 1.0.3 Az.PrivateDns {Add-AzPrivateDnsRecordConfig, Get-AzPrivateDnsRecordSet, Get-AzPrivateDnsVirtualNetworkLink, Get-AzPrivateDnsZone…}
Script 6.1.1 Az.RecoveryServices {Add-AzRecoveryServicesAsrReplicationProtectedItemDisk, Backup-AzRecoveryServicesBackupItem, Copy-AzRecoveryServicesVault, Disable-AzRecoveryServicesBackupAutoProtection…}
Script 1.6.0 Az.RedisCache {Export-AzRedisCache, Get-AzRedisCache, Get-AzRedisCacheFirewallRule, Get-AzRedisCacheKey…}
Script 1.1.0 Az.RedisEnterpriseCache {Export-AzRedisEnterpriseCache, Get-AzRedisEnterpriseCache, Get-AzRedisEnterpriseCacheDatabase, Get-AzRedisEnterpriseCacheKey…}
Script 1.0.3 Az.Relay {Get-AzRelayAuthorizationRule, Get-AzRelayHybridConnection, Get-AzRelayKey, Get-AzRelayNamespace…}
Script 1.1.0 Az.ResourceMover {Add-AzResourceMoverMoveResource, Get-AzResourceMoverMoveCollection, Get-AzResourceMoverMoveResource, Get-AzResourceMoverRequiredForResources…}
Script 6.4.1 Az.Resources {Export-AzResourceGroup, Export-AzTemplateSpec, Get-AzDenyAssignment, Get-AzDeployment…}
Script 1.3.0 Az.Security {Add-AzSecurityAdaptiveNetworkHardening, Add-AzSecuritySqlVulnerabilityAssessmentBaseline, Confirm-AzSecurityAutomation, Disable-AzIotSecurityAnalyticsAggregatedAlert…}
Script 3.0.0 Az.SecurityInsights {Get-AzSentinelAlertRule, Get-AzSentinelAlertRuleAction, Get-AzSentinelAlertRuleTemplate, Get-AzSentinelAutomationRule…}
Script 2.0.0 Az.ServiceBus {Add-AzServiceBusIPRule, Add-AzServiceBusVirtualNetworkRule, Get-AzServiceBusNamespace, Get-AzServiceBusOperation…}
Script 3.1.0 Az.ServiceFabric {Add-AzServiceFabricClientCertificate, Add-AzServiceFabricManagedClusterClientCertificate, Add-AzServiceFabricManagedNodeTypeVMExtension, Add-AzServiceFabricManagedNodeTypeVMSecret…}
Script 1.4.1 Az.SignalR {Get-AzSignalR, Get-AzSignalRKey, Get-AzSignalRUsage, New-AzSignalR…}
Script 4.1.0 Az.Sql {Add-AzSqlDatabaseToFailoverGroup, Add-AzSqlElasticJobStep, Add-AzSqlElasticJobTarget, Add-AzSqlInstanceKeyVaultKey…}
Script 1.1.0 Az.SqlVirtualMachine {Get-AzAvailabilityGroupListener, Get-AzSqlVM, Get-AzSqlVMGroup, New-AzAvailabilityGroupListener…}
Script 1.3.2 Az.StackHCI {Add-AzStackHCIVMAttestation, Disable-AzStackHCIAttestation, Disable-AzStackHCIRemoteSupport, Enable-AzStackHCIAttestation…}
Script 5.1.0 Az.Storage {Add-AzRmStorageContainerLegalHold, Add-AzStorageAccountManagementPolicyAction, Add-AzStorageAccountNetworkRule, Close-AzStorageFileHandle…}
Script 1.7.0 Az.StorageSync {Get-AzStorageSyncCloudEndpoint, Get-AzStorageSyncGroup, Get-AzStorageSyncServer, Get-AzStorageSyncServerEndpoint…}
Script 2.0.0 Az.StreamAnalytics {Get-AzStreamAnalyticsCluster, Get-AzStreamAnalyticsClusterStreamingJob, Get-AzStreamAnalyticsDefaultFunctionDefinition, Get-AzStreamAnalyticsFunction…}
Script 1.0.0 Az.Support {Get-AzSupportProblemClassification, Get-AzSupportService, Get-AzSupportTicket, Get-AzSupportTicketCommunication…}
Script 2.0.0 Az.Synapse {Add-AzSynapseDataFlowDebugSessionPackage, Add-AzSynapseTriggerSubscription, Clear-AzSynapseSqlPoolVulnerabilityAssessmentRuleBaseline, Convert-AzSynapseSqlPoolVulnerabilityAssessmentScan…}
Script 1.1.0 Az.TrafficManager {Add-AzTrafficManagerCustomHeaderToEndpoint, Add-AzTrafficManagerCustomHeaderToProfile, Add-AzTrafficManagerEndpointConfig, Add-AzTrafficManagerExpectedStatusCodeRange…}
Script 2.11.5 Az.Websites {Add-AzWebAppAccessRestrictionRule, Add-AzWebAppTrafficRouting, Edit-AzWebAppBackupConfiguration, Enter-AzWebAppContainerPSSession…}
```
### Error output
_No response_
Contributor guide
Research direction
Start at New-AzDataCollectionRule and trace how -RuleFile is converted into the Azure data collection rule request; compare the supplied dataFlows with the created resource payload. Done means transformKql and outputStream are retained for custom-table rules, with a regression test covering this JSON.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, powershell
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100