microsoft / microsoft/Dynamics-365-FastTrack-Implementation-Assets

Performance issue with incremntal GetLastChangedFeedFiles query

Open
#208 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
494
Forks
296
Avg merge
1d 10h
Merged PRs (30d)
5

Description

Hi,

We are facing a performance issue on GetLastChangedFeedFiles query. it's taking around 5mins to give the result, do you have any ideas to change the way it's written to improve the performance?

declare @Table_Name varchar(200) = '@{pipeline().parameters.TABLE_NAME}';
declare @edsname varchar(200) = (select top 1 name from sys.external_data_sources
where location like '%dfs.core.windows.net/dynamics365-financeandoperations%');

Declare @SQLQuery nvarchar(4000) = 'SELECT
r.filepath(1) AS [TABLE_NAME]
,max(r.filepath(2)) AS [LAST_UPDATED_CDC_FILE]
FROM OPENROWSET(
BULK ''ChangeFeed//.csv'',
DATA_SOURCE = ''' + @edsname+''',
FORMAT = ''CSV'',
PARSER_VERSION = ''2.0'',
FIRSTROW = 1)
With(FirstColum nvarchar(100))
as r
WHERE (r.filepath(1) = '''+ @Table_Name +''' OR ''' + @Table_Name+ ''' = '''')
GROUP BY
r.filepath(1)';

EXECUTE sp_executesql @SQLQuery;

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the GetLastChangedFeedFiles SQL query in the issue, running it with representative TABLE_NAME values and reviewing its execution plan and external file scan. Compare the current result and output with any revised query; done means the query returns the same data with a materially shorter runtime.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, sql
Domain
data-engineering, databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.