microsoft / microsoft/sql-server-samples
Daily ETL package skips data with incremental load
@jodebrui is already working on this.
Since Oct 21, 2016.
- Dominant language
- PowerShell
- Stars
- 11.2k
- Forks
- 9.1k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 14
Description
When refreshing data in the WWI sample databases, the SSIS package DailyETL can possible skip data when you're trying to incrementally load the data. (the documentation suggests to do a full load by reseeding the DWH first, but that's hardly a best practice)
For example, if I generate data for the Wide World Importers database using the stored procedure [DataLoadSimulation].[PopulateDataToCurrentDate], data is generated right until yesterday included. So if today is 19 September 2016, the last date will be the 18th of September 2016. When you run the SSIS package, the new cutoff time will be calculated as the current datetime - 5 minutes (e.g. 2016-9-19T21:50:00). So a certain point in time on the 19th of September 2016. All changes will be succesfully loaded into the DWH. When the ETL is done, the new cutoff time is written to the Integration.[ETL Cutoff] table in the DWH.
If at a later point in time I generate data again for the WWI database, data will also be generated for the 19th of September 2016. With the example of a cutoff time of 2016-9-19T21:50:00, you miss all changes from 2016-9-19T00:00:00 until 2016-9-19T21:50:00.
Instead of using a calculated cutoff time which is the same for all tables, it's maybe better to use the latest change date (e.g. LastEditedWhen) that can be found. For example:
DECLARE @NewCutfOff DATETIME = (SELECT MAX(LastEditedWhen) FROM Sales.Orders);
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.