MeltanoLabs / MeltanoLabs/tap-github

Incremental replication doesn't respect the current state

Open
#196 1 comment 1 reaction 1 assignee View on GitHub

@laurentS is already working on this.

Since May 11, 2023.

Dominant language
Python
Stars
20
Forks
60
Avg merge
20h 29m
Merged PRs (30d)
6

Description

The tap doesn't respect existing replication state by filter out data older than the replication key value in the state.

How to reproduce

Github tap configuration

  - name: tap-github-repos
    inherit_from: tap-github
    pip_url: git+https://github.com/MeltanoLabs/tap-github.git
    config:
      user_agent: ''
      start_date: '2023-01-01T00:00:00Z'
      searches:
      - name: All repos
        query: apache/*
    variant: meltanolabs
    select:
    - repositories.*
    metadata:
      repositories:
        replication-method: INCREMENTAL

Run a sync that produces 1000 (limit for the 'repositories' stream) records and a state record.

meltano run tap-github-repos target-jsonl

Run the same sync one more time

meltano run tap-github-repos target-jsonl

Result is there are 2000 records in the target json file and each record is fully duplicated.

The issue can be reproduced on the repositories stream.
I couldn't reproduce this on the issues stream.
I haven't tested other streams.

If Github APIs do not allow fetching data from a specific replication point (at least for the repositories stream) then the tap should filter those records instead of sending them down the pipeline.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.