microsoft / microsoft/Dynamics-365-FastTrack-Implementation-Assets
IncrementalExportSQL doesn't work
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 494
- Forks
- 296
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 5
Description
Hello,
I deployed cdmutil as Azure Function and evertything is working fine. Views are created in Synapse Serverless Pool, I can query data and ejoy life.
Anyhow, I want to transfer data from D365 data lake to Azure SQL Database. I downloaded ADF pipeline templates. Full export works fine but I have serious problem with incremental load.
If I try to trigger pipeline I recieve following error:
{
"errorCode": "2200",
"message": "Failure happened on 'Source' side. ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A database operation failed with the following error: 'Invalid object name 'd365._cdc_CustTable'.',Source=,''Type=System.Data.SqlClient.SqlException,Message=Invalid object name 'd365._cdc_CustTable'.,Source=.Net SqlClient Data Provider,SqlErrorNumber=208,Class=16,ErrorCode=-2146232060,State=1,Errors=[{Class=16,Number=208,State=1,Message=Invalid object name 'd365._cdc_CustTable'.,},],'",
"failureType": "UserError",
"target": "CDCCopy",
"details": []
}
CustTable is just example, I recieve same error for every table in my data lake. Views with preffix cdc don't exists in my Synapse Serverless Pool, I have regular named views.
Here is my configuration in Azure Function App:
DDLType: SynapseView
ParserVersion: 2.0 (I tried 1.0 with same effect)
ManifestURL: ends with Tables/Tables.manifest.cdm.json
Do you have any idea what's wrong?
Thank you in advance
Mateusz
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the SynapseToSQL_ADF pipeline templates and the Azure Function App configuration shown in the issue. Compare the incremental pipeline's expected cdc objects with the regular views generated in the Synapse Serverless Pool, then reproduce the CDCCopy failure. Done means the incremental pipeline can run for the configured tables without the invalid-object error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, sql
- Domain
- cloud, data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100