microsoft / microsoft/sql-server-samples

Daily ETL package skips data with incremental load

Open
#132 1 comment 0 reactions 1 assignee View on GitHub

@jodebrui is already working on this.

Since Oct 21, 2016.

Dominant language
PowerShell
Stars
11.2k
Forks
9.1k
Avg merge
2d 7h
Merged PRs (30d)
14

Description

When refreshing data in the WWI sample databases, the SSIS package DailyETL can possible skip data when you're trying to incrementally load the data. (the documentation suggests to do a full load by reseeding the DWH first, but that's hardly a best practice)

For example, if I generate data for the Wide World Importers database using the stored procedure [DataLoadSimulation].[PopulateDataToCurrentDate], data is generated right until yesterday included. So if today is 19 September 2016, the last date will be the 18th of September 2016. When you run the SSIS package, the new cutoff time will be calculated as the current datetime - 5 minutes (e.g. 2016-9-19T21:50:00). So a certain point in time on the 19th of September 2016. All changes will be succesfully loaded into the DWH. When the ETL is done, the new cutoff time is written to the Integration.[ETL Cutoff] table in the DWH.

If at a later point in time I generate data again for the WWI database, data will also be generated for the 19th of September 2016. With the example of a cutoff time of 2016-9-19T21:50:00, you miss all changes from 2016-9-19T00:00:00 until 2016-9-19T21:50:00.

Instead of using a calculated cutoff time which is the same for all tables, it's maybe better to use the latest change date (e.g. LastEditedWhen) that can be found. For example:

DECLARE @NewCutfOff DATETIME = (SELECT MAX(LastEditedWhen) FROM Sales.Orders);

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.