MeltanoLabs / MeltanoLabs/tap-github
Incremental replication doesn't respect the current state
@laurentS is already working on this.
Since May 11, 2023.
- Dominant language
- Python
- Stars
- 20
- Forks
- 60
- Avg merge
- 20h 29m
- Merged PRs (30d)
- 6
Description
The tap doesn't respect existing replication state by filter out data older than the replication key value in the state.
How to reproduce
Github tap configuration
- name: tap-github-repos
inherit_from: tap-github
pip_url: git+https://github.com/MeltanoLabs/tap-github.git
config:
user_agent: ''
start_date: '2023-01-01T00:00:00Z'
searches:
- name: All repos
query: apache/*
variant: meltanolabs
select:
- repositories.*
metadata:
repositories:
replication-method: INCREMENTAL
Run a sync that produces 1000 (limit for the 'repositories' stream) records and a state record.
meltano run tap-github-repos target-jsonl
Run the same sync one more time
meltano run tap-github-repos target-jsonl
Result is there are 2000 records in the target json file and each record is fully duplicated.
The issue can be reproduced on the repositories stream.
I couldn't reproduce this on the issues stream.
I haven't tested other streams.
If Github APIs do not allow fetching data from a specific replication point (at least for the repositories stream) then the tap should filter those records instead of sending them down the pipeline.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.