MeltanoLabs / MeltanoLabs/tap-github

State bookmarks should use ids instead of repo names

Open
#149 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
20
Forks
60
Avg merge
20h 29m
Merged PRs (30d)
6

Description

Following the same logic as what is described in #110 and the linked PR, the state bookmarks should use stable IDs for their context, as opposed to the current `org / repo`:
If a repo is renamed between 2 runs of the tap, the later run would ignore the bookmark, as the repo name would not match anymore. While not a disaster, it would essentially force a full resync of the history of the stream, which is a waste of time/energy/api quotas.

Example of a state bookmark right now:
```json
{
"context": { "org": "scipy", "repo": "scipy" }, // this should use a stable id
"replication_key": "updated_at",
"replication_key_value": "2022-05-06T14:29:18Z",
"replication_key_signpost": "2022-06-22T09:07:14.047100+00:00",
"starting_replication_value": "2022-05-06T14:29:18Z",
"progress_markers": {
"Note": "Progress is not resumable if interrupted.",
"replication_key": "updated_at",
"replication_key_value": "2022-06-22T07:55:32Z"
}
},
```

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing issue #110 and its linked pull request, then locate the state bookmark serialization and matching logic in the tap. Confirm how repository rename handling currently behaves; done means bookmarks use a stable repository identifier and a renamed repository can resume without a full history resync.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.