microsoft / microsoft/sql-server-samples

Daily ETL package skips data with incremental load

Aperta
#132 1 commento 0 reazioni 1 assegnatario Vedi su GitHub

@jodebrui ci sta già lavorando.

Dal 21/10/2016.

Lingua principale
PowerShell
Stelle
11.2k
Fork
9.1k
Merge medio
2g 7h
PR unite (30g)
14

Descrizione

When refreshing data in the WWI sample databases, the SSIS package DailyETL can possible skip data when you're trying to incrementally load the data. (the documentation suggests to do a full load by reseeding the DWH first, but that's hardly a best practice)

For example, if I generate data for the Wide World Importers database using the stored procedure [DataLoadSimulation].[PopulateDataToCurrentDate], data is generated right until yesterday included. So if today is 19 September 2016, the last date will be the 18th of September 2016. When you run the SSIS package, the new cutoff time will be calculated as the current datetime - 5 minutes (e.g. 2016-9-19T21:50:00). So a certain point in time on the 19th of September 2016. All changes will be succesfully loaded into the DWH. When the ETL is done, the new cutoff time is written to the Integration.[ETL Cutoff] table in the DWH.

If at a later point in time I generate data again for the WWI database, data will also be generated for the 19th of September 2016. With the example of a cutoff time of 2016-9-19T21:50:00, you miss all changes from 2016-9-19T00:00:00 until 2016-9-19T21:50:00.

Instead of using a calculated cutoff time which is the same for all tables, it's maybe better to use the latest change date (e.g. LastEditedWhen) that can be found. For example:

DECLARE @NewCutfOff DATETIME = (SELECT MAX(LastEditedWhen) FROM Sales.Orders);

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.