microsoft / microsoft/Dynamics-365-FastTrack-Implementation-Assets
Synapse link with incremental update timeout
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 494
- Forks
- 296
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 5
Description
Hi,
I deployed the solution found at the "https://github.com/microsoft/Dynamics-365-FastTrack-Implementation-Assets/tree/master/Analytics/DataverseLink/DataIntegration" to get csv files exported by Synapse Link into our Dedicated Pool DB.
The problem is that for large tables, the query on the source interface (Synapse SQL Serverless) return error due to timeout, 30 minutes on Synapse SQL Serverless. Also, if the pipeline return errors, the involved tables are not reprocessed on the next run, cause the controtable on the destination db contains the table entry with end date set to null, I must delete the table entry to allow reprocess by pipeline, however the problem rise again cause table size.
I've found tables with many records also, but they have less columns.
The source CSV are stored on a datalake and they are gained by Dynamics 365 F&O.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the DataIntegration solution linked in the issue, focusing first on the Synapse SQL Serverless source query and the destination control table used by the pipeline. Reproduce the behavior with a large Dynamics 365 F&O table and inspect what happens after a timeout. Done means large-table processing avoids the reported timeout or handles it appropriately, and failed tables are eligible for the next pipeline run without manual deletion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, sql
- Domain
- cloud, data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100