Azure / Azure/azure-cli

SubType value ParquetReadSettings has no mapping. Synapse workspace Pipeline.

Open
#28,606 4 comments 0 reactions 0 assignees View on GitHub
act-codegen-extensibility-squad Auto-Assign bug customer-reported Service Attention Synapse
Dominant language
Python
Stars
4.6k
Forks
3.5k
Avg merge
3d 2h
Merged PRs (30d)
60

Description

### Describe the bug

Issue: The property ParquetReadSettings is missed in the JSON Synapse Pipeline definition after doing a CLI deployment process in his DevOps.

Error from Synapse after deployment

It is showed fine in the JSON definition but it removed this property after the deployment.

Before CLI
![image](https://github.com/Azure/azure-cli/assets/136427946/3dd1f18c-e0c1-4e9e-b79e-6ea2f2e8eb54)


After CLI.
![image](https://github.com/Azure/azure-cli/assets/136427946/66c4ff2e-1d96-4700-a452-7c675d568d97)



Troubleshoot.

Checking on CLI console we did see a warning during the deployment.

![image](https://github.com/Azure/azure-cli/assets/136427946/f4de7e76-1b62-46c0-97cc-357b4befb882)


This completes but after it happens the JSON in the Synapse changed removing this property in it.

I was able to reproduce the issue using the metadata activity where is this property is located.

![image](https://github.com/Azure/azure-cli/assets/136427946/d2942963-3d2e-4020-b296-04221d4b977a)

Steps to reproduce:
Create a pipeline with a metadata activity
Linked sevice to ADLS2
Parquet Data Set
Field List Exist
Save JSON as a JsonFIle in a local folder in the PC
run CLI command az synapse pipeline update --workspace-name synapseWorksapceName --name CLI_Test --file "@C:/Users/jonathanh/Documents/SYnapseCLITest/CLI_Pipeline.json"
A warning will be thrown
check the Pipeline in the Synapse UI It will show a JSON parse error
Verify the JSON definition, the ParqueReadSettinng porperty has been _removed.___

### Related command

az synapse pipeline update --workspace-name synapseWorkspaceName --name CLI_Test --file "@C:/Users/jonathanh/Documents/SYnapseCLITest/CLI_Pipeline.json"

### Errors

![image](https://github.com/Azure/azure-cli/assets/136427946/c9640077-023a-483b-a386-677f2c6de913)

### Issue script & Debug output

{
"name": "CLI_Test",
"properties": {
"activities": [
{
"name": "Wait1",
"type": "Wait",
"dependsOn": [
{
"activity": "Get Metadata1",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"waitTimeInSeconds": 1
}
},
{
"name": "Get Metadata1",
"type": "GetMetadata",
"dependsOn": [],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"dataset": {
"referenceName": "Parquet1",
"type": "DatasetReference"
},
"fieldList": [
"exists"
],
"storeSettings": {
"type": "AzureBlobFSReadSettings",
"enablePartitionDiscovery": false
},
"formatSettings": {
**"type": "ParquetReadSettings"**
}
}
}
],
"annotations": []
}
}

### Expected behavior

{
"name": "CLI_Test",
"properties": {
"activities": [
{
"name": "Wait1",
"type": "Wait",
"dependsOn": [
{
"activity": "Get Metadata1",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"waitTimeInSeconds": 1
}
},
{
"name": "Get Metadata1",
"type": "GetMetadata",
"dependsOn": [],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"dataset": {
"referenceName": "Parquet1",
"type": "DatasetReference"
},
"fieldList": [
"exists"
],
"storeSettings": {
"type": "AzureBlobFSReadSettings",
"enablePartitionDiscovery": false
},
**"formatSettings": { }**
}
}
],
"annotations": []
}
}

### Environment Summary

azure-cli 2.58.0

core 2.58.0
telemetry 1.1.0

Dependencies:
msal 1.26.0
azure-mgmt-resource 23.1.0b2

Python location 'C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe'
Extensions directory 'C:\Users\jonathanh\.azure\cliextensions'

Python (Windows) 3.11.7 (tags/v3.11.7:fa7a6f2, Dec 4 2023, 19:24:49) [MSC v.1937 64 bit (AMD64)]

Legal docs and information: aka.ms/AzureCliLegal

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.