dlt-hub / dlt-hub/dlt

[documentation] Incremental loads which extract data from a source database table using a timestamp column may miss rows

Open
#2,269 6 comments 0 reactions 0 assignees View on GitHub
documentation
Dominant language
Python
Stars
5.9k
Forks
605
Avg merge
1d 14h
Merged PRs (30d)
38

Description

### Proposed solution
Update incremental documentation to hint how to use `lag` feature to acquire records created in certain time window - on systems with frequent and concurrent updates coming from the application layer

see:
https://github.com/dlt-hub/dlt/issues/2269#issuecomment-2648849631

### dlt version

1.5.0

### Describe the problem

Assume the source database has a table called `events` where rows are only ever inserted (never updated or deleted) with a timestamp column called `created`

The snippet of code below is intended to load new rows into the destination table incrementally

```
source_1 = sql_database(credentials).with_resources("events")
source_1.events.apply_hints(incremental=dlt.sources.incremental("created"))

info = pipeline.run(source_1, write_disposition="append")
```

At the start of a pipeline run, assume the maximum value of the events.created = '2025-02-05 02:30:00'

It appears that dlt stores this value ('2025-02-05 02:30:00') as a 'high water mark' to be used in the next pipeline run. However, on a busy application / database, it's quite possible that a new row is committed to the database AFTER the pipeline started with an earlier `events.created` value (of say, '2025-02-05 02:29:59').

As a result, this row would not be included in the subsequent pipeline run, since it appears it applies a filter SELECT * FROM events where created >

I would assume the same kind of issue occurs when using an auto-increment column

### Expected behavior

Best thing I can suggest is that dlt allows developers to access / modify the high-water-mark value to allow for such lags / latencies which can occur from the point at which an application assigns a timestamp to a column and the time the database actually commits that value. This latency can vary from system to system.

A side effect of this is that consecutive pipeline runs may fetch the same subset of rows. This means the destination table will contain duplicates (unless dlt takes measures to deduplicate - for example :

```
INSERT into destination.table as tgt
where not exists
(select 1 from destination.table as tgt1
where tgt1. = tgt.)
```

### Steps to reproduce

1) create a database table in the source database

2) manually insert events into to the table where max event.created =

3) run the dlt pipeline

4) manually insert new events into to the source database table where :
- some events have `event.created > `
- some events have `event.created < `

5) run the dlt pipeline

6) check the corresponding destination database table - the rows in the source having `event.created < ` will not be present in the destination

### Operating system

macOS

### Runtime environment

Local

### Python version

3.10

### dlt data source

postgresql

### dlt destination

_No response_

### Other deployment details

postgres

### Additional information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with dlt's incremental loading documentation and the `sql_database`/`incremental` entry points described in the issue. Check how the `lag` feature is documented, then add guidance for timestamp-based loads on busy source databases; done means the documented workflow explains the time-window tradeoff and possible duplicate rows.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
data-engineering, databases, documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.