Azure / Azure/Azure-DataFactory
Data flow: toBase64 adds break lines after 76th character in specific ADF location
- Dominant language
- PowerShell
- Stars
- 529
- Forks
- 623
- PR merge metrics
- No merged PRs in 30d
Description
Hi!
Recently we've got unexpected outputs from ADF pipelines with no prior changes to the source code.
Suddenly, the toBase64 function in data flow activities began to add break lines (\r\n) after the 76th character for output values.
During the issue investigation, we had an assumption that it could be some Virtual Machine or SDK's specific thing, based on this we tried to play with the locations of ADF, and it worked out:
If the location of ADF is West Europe then all good and break lines aren't added:

if we change it to UK West then suddenly break lines start to appear:

I am attaching the ARM template of ADF, containing the pipeline that uses the dataflow that converts input into base64 and the blob files used as input and output.
The payload is simple in JSON format and contains a single example property "ContactEmail".
You will also need a Storage Account with container "testbase64" and blobs: "files/test1" that would be as input and "output/test1output" that would be as output.
Steps to reproduce:
1) Create the Storage Account
1.1) Create container "testbase64"
1.2) Create blobs "files/test1", "output/test1output" in that container
2) Create 2 ADFs:
2.1) Location = West Europe
2.2) Location = UK West
3) Import ARM template into created ADFs
4) Put a long enough value into "test1" file so that it exceeds 76 chars in output, like:
{"ContactEmail":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"}
5) Run pipelines and compare results
As a solution, we are considering redeploying ADF into a different location.
If you have some other options on your mind, we'd appreciate it.
Thanks in advance!
Files:
[ARM Template and Blobs.zip](https://github.com/Azure/Azure-DataFactory/files/15107851/ARM.Template.and.Blobs.zip)
Contributor guide
Assessment
This issue has not been assessed yet.