[documentation] Incremental loads which extract data from a source database table using a timestamp column may miss rows
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 605
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
### Proposed solution
Update incremental documentation to hint how to use `lag` feature to acquire records created in certain time window - on systems with frequent and concurrent updates coming from the application layer
see:
https://github.com/dlt-hub/dlt/issues/2269#issuecomment-2648849631
### dlt version
1.5.0
### Describe the problem
Assume the source database has a table called `events` where rows are only ever inserted (never updated or deleted) with a timestamp column called `created`
The snippet of code below is intended to load new rows into the destination table incrementally
```
source_1 = sql_database(credentials).with_resources("events")
source_1.events.apply_hints(incremental=dlt.sources.incremental("created"))
info = pipeline.run(source_1, write_disposition="append")
```
At the start of a pipeline run, assume the maximum value of the events.created = '2025-02-05 02:30:00'
It appears that dlt stores this value ('2025-02-05 02:30:00') as a 'high water mark' to be used in the next pipeline run. However, on a busy application / database, it's quite possible that a new row is committed to the database AFTER the pipeline started with an earlier `events.created` value (of say, '2025-02-05 02:29:59').
As a result, this row would not be included in the subsequent pipeline run, since it appears it applies a filter SELECT * FROM events where created >
I would assume the same kind of issue occurs when using an auto-increment column
### Expected behavior
Best thing I can suggest is that dlt allows developers to access / modify the high-water-mark value to allow for such lags / latencies which can occur from the point at which an application assigns a timestamp to a column and the time the database actually commits that value. This latency can vary from system to system.
A side effect of this is that consecutive pipeline runs may fetch the same subset of rows. This means the destination table will contain duplicates (unless dlt takes measures to deduplicate - for example :
```
INSERT into destination.table as tgt
where not exists
(select 1 from destination.table as tgt1
where tgt1. = tgt.)
```
### Steps to reproduce
1) create a database table in the source database
2) manually insert events into to the table where max event.created =
3) run the dlt pipeline
4) manually insert new events into to the source database table where :
- some events have `event.created > `
- some events have `event.created < `
5) run the dlt pipeline
6) check the corresponding destination database table - the rows in the source having `event.created < ` will not be present in the destination
### Operating system
macOS
### Runtime environment
Local
### Python version
3.10
### dlt data source
postgresql
### dlt destination
_No response_
### Other deployment details
postgres
### Additional information
_No response_
Contributor guide
Research direction
Start with dlt's incremental loading documentation and the `sql_database`/`incremental` entry points described in the issue. Check how the `lag` feature is documented, then add guidance for timestamp-based loads on busy source databases; done means the documented workflow explains the time-window tradeoff and possible duplicate rows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- data-engineering, databases, documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100