Azure / Azure/Azure-DataFactory

Complex Array Typing Lost on Rule Base Mapping Select

Open
#558 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
PowerShell
Stars
529
Forks
623
PR merge metrics
No merged PRs in 30d

Description

Hi,

I have run into an issue where if I use a rule based mapping in a select to elevate properties up a scope, the typing information for a complex or complex array are lost:

While in this example there are only 3 properties, my actual workflow I have over 600 properties which is why I was trying to use the rule-based mapping.

I am specifically dealing with:
- the Data Factory UI
- a flowlet
- a json file
- a select schema modifier
- a rule based mapping to elevate sub properties to a global scope

example json:
```json
{
"thing": {
"id": 1234,
"complexArray": [
{"id": 223, "name": "test 1"},
{"id": 222, "name": "test 2"}
],
"complex": {
"id": 232,
"name": "test 3",
"type": "Complex"
}
}
}
```

selecting this json file as a source brings in the types correctly no problem:
![image](https://user-images.githubusercontent.com/13630698/228258013-18cb8a84-16e7-4649-b962-6ef46ad631e6.png)

if I then run a select because I want to unwrap the "thing" property and elevate its properties to the main scope, I lose the typing information which causes an error.

Select rule based mapping:
![image](https://user-images.githubusercontent.com/13630698/228258393-6eece8c4-b4bd-4e6c-a5a3-1805296ba9b0.png)

which leads to `any` instead of the `complex` type:
![image](https://user-images.githubusercontent.com/13630698/228258531-1e198a5d-1aa3-41a0-848e-b399dceadea6.png)

and then causes me an error when I try to have this as an output in a flowlet:
![image](https://user-images.githubusercontent.com/13630698/228258805-42133ffa-dbcf-4cde-97de-e336cf7fb847.png)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.