Azure / Azure/Azure-DataFactory

How to transform complex json files using ADF

Open
#359 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
PowerShell
Stars
529
Forks
623
PR merge metrics
No merged PRs in 30d

Description

I want to transform multiple complex JSON files into one complex JSON file using Azure Data Factory dataflow.

The multiple complex input JSON files are in the following format:
```json
{
"creationDate": "2022-01-19T17:00:17Z",
"count": 2,
"data": [
{
"id": "",
"name": "Project A",
"url": "",
"state": "Open",
"revision": 1,
"visibility": "private",
"lastUpdateTime": "2019-09-23T08:44:45.103Z"
},
{
"id": "",
"name": "Project B",
"url": "",
"state": "Done",
"revision": 1,
"visibility": "private",
"lastUpdateTime": "2019-12-31T09:38:49.16Z"
}
]
}
```

We want to transform those files to one single json file in the format:
```json
[
{
"date": "2022-01-14",
"count": 2,
"projects": [
{
"name": "Project A",
"state": "",
"lastUpdateTime": ""
},
{
"name": "Project B",
"state": "",
"lastUpdateTime": ""
}
]
},
{
"date": "2022-01-17",
"count": 3,
"projects": [
{
"name": "Project A",
"state": "",
"lastUpdateTime": ""
},
{
"name": "Project B",
"state": "",
"lastUpdateTime": ""
},
{
"name": "Project C",
"state": "",
"lastUpdateTime": ""
}
]
}
]
```

We were using the derived column with the expression `@(name=data.name, state=data.state)`.
[![enter image description here][1]][1]

Can someone help us how to do this? We tried a lot of things like derived column, first flattening but we can't get it as we like...

Thanks!

[1]: https://i.stack.imgur.com/GubnH.png

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.