MeltanoLabs / MeltanoLabs/tap-github
State bookmarks should use ids instead of repo names
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 20
- Forks
- 60
- Avg merge
- 20h 29m
- Merged PRs (30d)
- 6
Description
Following the same logic as what is described in #110 and the linked PR, the state bookmarks should use stable IDs for their context, as opposed to the current `org / repo`:
If a repo is renamed between 2 runs of the tap, the later run would ignore the bookmark, as the repo name would not match anymore. While not a disaster, it would essentially force a full resync of the history of the stream, which is a waste of time/energy/api quotas.
Example of a state bookmark right now:
```json
{
"context": { "org": "scipy", "repo": "scipy" }, // this should use a stable id
"replication_key": "updated_at",
"replication_key_value": "2022-05-06T14:29:18Z",
"replication_key_signpost": "2022-06-22T09:07:14.047100+00:00",
"starting_replication_value": "2022-05-06T14:29:18Z",
"progress_markers": {
"Note": "Progress is not resumable if interrupted.",
"replication_key": "updated_at",
"replication_key_value": "2022-06-22T07:55:32Z"
}
},
```
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing issue #110 and its linked pull request, then locate the state bookmark serialization and matching logic in the tap. Confirm how repository rename handling currently behaves; done means bookmarks use a stable repository identifier and a renamed repository can resume without a full history resync.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100