microsoft / microsoft/Dynamics-365-FastTrack-Implementation-Assets

Error In the DataLakeToSQLCopy Pipeline

Open
#241 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
494
Forks
296
Avg merge
1d 10h
Merged PRs (30d)
5

Description

Hello @jiyada

I have been working on implementing the Copy from ADLS into Azure SQL DB Solution as described here https://github.com/microsoft/Dynamics-365-FastTrack-Implementation-Assets/blob/master/Analytics/CDMUtilSolution/readme_datacopy.md

However, the DataLakeToSQLCopy (Dataflow part) keeps failing and outputs the following error:
Error
DataLakeToSQL_CDMToSQL_Dataflows
{"message":"Job failed due to reason: com.microsoft.dataflow.broker.MissingRequiredPropertyException: account is a required property for Source_Datalake_LinkedService1c25c14e035c4852bf29a4c7ea3fe8da.\ncom.microsoft.dataflow.broker.PropertyNotFoundException: Could not extract value from Source_Datalake_LinkedService1c25c14e035c4852bf29a4c7ea3fe8da. Details:","failureType":"UserError","target":"DataLakeToSQL_CDMToSQL_Dataflows","errorCode":"DFExecutorUserError"}

I tried debugging the issue but could not find this property "account".

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with readme_datacopy.md and the DataLakeToSQL_CDMToSQL_Dataflows dataflow configuration. Trace Source_Datalake_LinkedService1c25c14e035c4852bf29a4c7ea3fe8da and reproduce the reported MissingRequiredPropertyException. Done means the DataLakeToSQLCopy pipeline runs without the missing account-property error.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, sql
Domain
cloud, data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.