Azure / Azure/azure-powershell

Unable to deserialize the response - Get-AzDataFactoryV2Pipeline

Open
#20,722 12 comments 0 reactions 0 assignees View on GitHub
act-codegen-extensibility-squad bug customer-reported Data Factory needs-team-attention Service Attention
Dominant language
C#
Stars
4.8k
Forks
4.3k
Avg merge
2d 17h
Merged PRs (30d)
51

Description

### Description

`Get-AzDataFactoryV2Pipeline -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName `

I've gone through each pipeline individually and located the pipeline which cannot be deserialized. I removed code, one item at a time until I identified the particular item causing the issue.

In one of the copy data activities I have set the Source "Read behavior" to dynamic content, below. The pipeline has been running in production for months now and the code works. It appears that Get-AzDataFactoryV2Pipeline does not support deserializing this logic. Any ideas how to get this supported/corrected?

`@if(equals(pipeline().parameters.IsIncremental, bool(true)), 'queryAll', 'query')`

### Issue script & Debug output

```PowerShell
DEBUG: 8:07:00 AM - GetAzureDataFactoryPipelineCommand begin processing with ParameterSet 'ByFactoryName'.
DEBUG: 8:07:01 AM - using account id 'xxxxxx'...
DEBUG: 8:07:01 AM - [ConfigManager] Got nothing from [DisplayBreakingChangeWarning], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: [Common.Authentication]: Authenticating using Account: 'xxxxxx', environment: 'AzureCloud', tenant: 'xxxxxxx'
DEBUG: 8:07:01 AM - [SilentAuthenticator] Calling SharedTokenCacheCredential.GetTokenAsync - TenantId:'xxxxxxxxxxx', Scopes:'https://management.core.windows.net//.default', Authori
tyHost:'https://login.microsoftonline.com/', UserId:'xxxx'
DEBUG: SharedTokenCacheCredential.GetToken invoked. Scopes: [ https://management.core.windows.net//.default ] ParentRequestId:
DEBUG: False MSAL 4.49.1.0 MSAL.Desktop 4.8 or later Windows 10 Enterprise [2023-01-20 13:07:01Z - 932f60b7-e23f-47ee-aa20-96442fb21d97] IsLegacyAdalCacheEnabled: yes
DEBUG: False MSAL 4.49.1.0 MSAL.Desktop 4.8 or later Windows 10 Enterprise [2023-01-20 13:07:01Z - 932f60b7-e23f-47ee-aa20-96442fb21d97] [Region discovery] Not using a regional authority.
DEBUG: False MSAL 4.49.1.0 MSAL.Desktop 4.8 or later Windows 10 Enterprise [2023-01-20 13:07:01Z - 932f60b7-e23f-47ee-aa20-96442fb21d97] [Region discovery] Not using a regional authority.
DEBUG: False MSAL 4.49.1.0 MSAL.Desktop 4.8 or later Windows 10 Enterprise [2023-01-20 13:07:01Z - 932f60b7-e23f-47ee-aa20-96442fb21d97] IsLegacyAdalCacheEnabled: yes
DEBUG: False MSAL 4.49.1.0 MSAL.Desktop 4.8 or later Windows 10 Enterprise [2023-01-20 13:07:01Z - 932f60b7-e23f-47ee-aa20-96442fb21d97] IsLegacyAdalCacheEnabled: yes
DEBUG: False MSAL 4.49.1.0 MSAL.Desktop 4.8 or later Windows 10 Enterprise [2023-01-20 13:07:01Z] [WamBroker] WAM supported OS.
DEBUG: False MSAL 4.49.1.0 MSAL.Desktop 4.8 or later Windows 10 Enterprise [2023-01-20 13:07:01Z] [WamBroker] ListWindowsWorkAndSchoolAccounts option was not enabled.
DEBUG: False MSAL 4.49.1.0 MSAL.Desktop 4.8 or later Windows 10 Enterprise [2023-01-20 13:07:01Z - 6a4799d6-e974-4d41-a38b-526589ac3da5] [Region discovery] Not using a regional authority.
DEBUG: False MSAL 4.49.1.0 MSAL.Desktop 4.8 or later Windows 10 Enterprise [2023-01-20 13:07:01Z] Found 1 cache accounts and 0 broker accounts
DEBUG: False MSAL 4.49.1.0 MSAL.Desktop 4.8 or later Windows 10 Enterprise [2023-01-20 13:07:01Z] Returning 1 accounts
DEBUG: False MSAL 4.49.1.0 MSAL.Desktop 4.8 or later Windows 10 Enterprise [2023-01-20 13:07:01Z - 3cb18f39-6308-4ee1-a3f0-0b28459932db] MSAL MSAL.Desktop with assembly version '4.49.1.0'. CorrelationId(3c
b18f39-6308-4ee1-a3f0-0b28459932db)
DEBUG: False MSAL 4.49.1.0 MSAL.Desktop 4.8 or later Windows 10 Enterprise [2023-01-20 13:07:01Z - 3cb18f39-6308-4ee1-a3f0-0b28459932db] === AcquireTokenSilent Parameters ===
DEBUG: False MSAL 4.49.1.0 MSAL.Desktop 4.8 or later Windows 10 Enterprise [2023-01-20 13:07:01Z - 3cb18f39-6308-4ee1-a3f0-0b28459932db] LoginHint provided: False
DEBUG: False MSAL 4.49.1.0 MSAL.Desktop 4.8 or later Windows 10 Enterprise [2023-01-20 13:07:01Z - 3cb18f39-6308-4ee1-a3f0-0b28459932db] Account provided: True
DEBUG: False MSAL 4.49.1.0 MSAL.Desktop 4.8 or later Windows 10 Enterprise [2023-01-20 13:07:01Z - 3cb18f39-6308-4ee1-a3f0-0b28459932db] ForceRefresh: False
DEBUG: False MSAL 4.49.1.0 MSAL.Desktop 4.8 or later Windows 10 Enterprise [2023-01-20 13:07:01Z - 3cb18f39-6308-4ee1-a3f0-0b28459932db]
=== Request Data ===
Authority Provided? - True
Scopes - https://management.core.windows.net//.default
Extra Query Params Keys (space separated) -
ApiId - AcquireTokenSilent
IsConfidentialClient - False
SendX5C - False
LoginHint ? False
IsBrokerConfigured - True
HomeAccountId - False
CorrelationId - 3cb18f39-6308-4ee1-a3f0-0b28459932db
UserAssertion set: False
LongRunningOboCacheKey set: False
Region configured:

DEBUG: False MSAL 4.49.1.0 MSAL.Desktop 4.8 or later Windows 10 Enterprise [2023-01-20 13:07:01Z - 3cb18f39-6308-4ee1-a3f0-0b28459932db] === Token Acquisition (SilentRequest) started:
Scopes: https://management.core.windows.net//.default
Authority Host: login.microsoftonline.com
DEBUG: False MSAL 4.49.1.0 MSAL.Desktop 4.8 or later Windows 10 Enterprise [2023-01-20 13:07:01Z - 3cb18f39-6308-4ee1-a3f0-0b28459932db] [Region discovery] Not using a regional authority.
DEBUG: False MSAL 4.49.1.0 MSAL.Desktop 4.8 or later Windows 10 Enterprise [2023-01-20 13:07:01Z - 3cb18f39-6308-4ee1-a3f0-0b28459932db] Access token is not expired. Returning the found cache entry. [Curre
nt time (01/20/2023 13:07:01) - Expiration Time (01/20/2023 14:11:57 +00:00) - Extended Expiration Time (01/20/2023 14:11:57 +00:00)]
DEBUG: False MSAL 4.49.1.0 MSAL.Desktop 4.8 or later Windows 10 Enterprise [2023-01-20 13:07:01Z - 3cb18f39-6308-4ee1-a3f0-0b28459932db] Returning access token found in cache. RefreshOn exists ? False
DEBUG: False MSAL 4.49.1.0 MSAL.Desktop 4.8 or later Windows 10 Enterprise [2023-01-20 13:07:01Z - 3cb18f39-6308-4ee1-a3f0-0b28459932db] [Region discovery] Not using a regional authority.
DEBUG: False MSAL 4.49.1.0 MSAL.Desktop 4.8 or later Windows 10 Enterprise [2023-01-20 13:07:01Z - 3cb18f39-6308-4ee1-a3f0-0b28459932db]
=== Token Acquisition finished successfully:
DEBUG: False MSAL 4.49.1.0 MSAL.Desktop 4.8 or later Windows 10 Enterprise [2023-01-20 13:07:01Z - 3cb18f39-6308-4ee1-a3f0-0b28459932db] AT expiration time: 1/20/2023 2:11:57 PM +00:00, scopes: https://ma
nagement.core.windows.net//user_impersonation https://management.core.windows.net//.default. source: Cache
DEBUG: SharedTokenCacheCredential.GetToken succeeded. Scopes: [ https://management.core.windows.net//.default ] ParentRequestId: ExpiresOn: 2023-01-20T14:11:57.0000000+00:00
DEBUG: [Common.Authentication]: Received token with LoginType 'User', Tenant: 'xxxxx', UserId: 'xxxx'
DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
GET

Absolute Uri:
https://management.azure.com/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.DataFactory/factories/xxx/pipelines/xxx?api-version=2018-06-01

Headers:
x-ms-client-request-id : aa96fd89-320a-4e64-a59b-4461a3e37156
accept-language : en-US

Body:

DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
OK

DEBUG: 8:06:13 AM - [ConfigManager] Got nothing from [EnableDataCollection], Module = [], Cmdlet = []. Returning default value [True].
Get-AzDataFactoryV2Pipeline : Unable to deserialize the response.
At D:\_Code\Samples\PowerShell\Az Login.ps1:5 char:1
+ Get-AzDataFactoryV2Pipeline -ResourceGroupName "rg-flexwarehouse-dev" ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Get-AzDataFactoryV2Pipeline], SerializationException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.DataFactoryV2.GetAzureDataFactoryPipelineCommand

DEBUG: 8:06:15 AM - [ConfigManager] Got nothing from [DisplayBreakingChangeWarning], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: AzureQoSEvent: Module: Az.DataFactory:1.16.11; CommandName: Get-AzDataFactoryV2Pipeline; PSVersion: 5.1.19041.2364; IsSuccess: False; Duration: 00:00:05.6280744;
Exception: Unable to deserialize the response.;
DEBUG: 8:06:15 AM - GetAzureDataFactoryPipelineCommand end processing.
```

### Environment data

```PowerShell
Name Value
---- -----
PSVersion 5.1.19041.2364
PSEdition Desktop
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}
BuildVersion 10.0.19041.2364
CLRVersion 4.0.30319.42000
WSManStackVersion 3.0
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
```

### Module versions

```PowerShell
ModuleType Version Name ExportedCommands
---------- ------- ---- ----------------
Script 2.11.1 Az.Accounts {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-AzDefault...}
Script 1.16.11 Az.DataFactory {Add-AzDataFactoryV2DataFlowDebugSessionPackage, Add-AzDataFactoryV2TriggerSubscription, Get-AzDataFactory, Get-AzDataFactoryActivityWindow...}
```

### Error output

```PowerShell
HistoryId: 1

Message : Unable to deserialize the response.
StackTrace : at Microsoft.Azure.Management.DataFactory.PipelinesOperations.d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Management.DataFactory.PipelinesOperationsExtensions.d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Management.DataFactory.PipelinesOperationsExtensions.ListByFactory(IPipelinesOperations operations, String resourceGroupName, String factoryName)
at Microsoft.Azure.Commands.DataFactoryV2.DataFactoryClient.ListPipelines(AdfEntityFilterOptions filterOptions)
at Microsoft.Azure.Commands.DataFactoryV2.DataFactoryClient.FilterPSPipelines(AdfEntityFilterOptions filterOptions)
at Microsoft.Azure.Commands.DataFactoryV2.GetAzureDataFactoryPipelineCommand.ExecuteCmdlet()
at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()
Exception : Microsoft.Rest.SerializationException
InvocationInfo : {Get-AzDataFactoryV2Pipeline}
Line : Get-AzDataFactoryV2Pipeline -ResourceGroupName "rg-flexwarehouse-dev" -DataFactoryName "datafactorySalesforceLoader" -debug

Position : At D:\_Code\Samples\PowerShell\Az Login.ps1:5 char:1
+ Get-AzDataFactoryV2Pipeline -ResourceGroupName "rg-flexwarehouse-dev" ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId : 1

Message : Error reading string. Unexpected token: StartObject. Path 'readBehavior', line 3970, position 33.
StackTrace : at Newtonsoft.Json.JsonReader.ReadAsString()
at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.Linq.JToken.ToObject(Type objectType, JsonSerializer jsonSerializer)
at Microsoft.Rest.Serialization.PolymorphicDeserializeJsonConverter`1.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.Linq.JToken.ToObject(Type objectType, JsonSerializer jsonSerializer)
at Microsoft.Rest.Serialization.PolymorphicDeserializeJsonConverter`1.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String
id)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract
containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.Linq.JToken.ToObject(Type objectType, JsonSerializer jsonSerializer)
at Microsoft.Rest.Serialization.TransformationJsonConverter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String
id)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract
containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract,
JsonProperty containerProperty, JsonReader reader, Object target)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract
containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract
containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Microsoft.Rest.Serialization.SafeJsonConvert.DeserializeObject[T](String json, JsonSerializerSettings settings)
at Microsoft.Azure.Management.DataFactory.PipelinesOperations.d__5.MoveNext()
Exception : Newtonsoft.Json.JsonReaderException
InvocationInfo : {Get-AzDataFactoryV2Pipeline}
Line : Get-AzDataFactoryV2Pipeline -ResourceGroupName "rg-flexwarehouse-dev" -DataFactoryName "datafactorySalesforceLoader" -debug

Position : At D:\_Code\Samples\PowerShell\Az Login.ps1:5 char:1
+ Get-AzDataFactoryV2Pipeline -ResourceGroupName "rg-flexwarehouse-dev" ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId : 1
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.