singer-io / singer-io/tap-github
Suggestion: mark "starting repository" in bookmark data
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 76
- Forks
- 86
- PR merge metrics
- No merged PRs in 30d
Description
If you had a very large number of repositories being fetched, say 50-100, it may be that you can never get through everything in an hour before you hit your 5000 request limit. I haven't hit this issue myself yet (well, I'd want to see where we get to with #43), so it's a little hypothetical, but I think it could become an issue in a future use-case (I'm likely to want to track about 80+ repos in my full implementation)
In such as a case, it would be absolutely fine to try again with the next fetch in an hour's time, but I would want to pick up where I left off.
So let's say I was tracking these 3 repos:
- silverstripe/silverstripe-framework
- silverstripe/silverstripe-cms
- silverstripe/silverstripe-admin
- silverstripe/silverstripe-versioned
- silverstripe/silverstripe-assets
And in my import I got
- silverstripe/silverstripe-framework (finished)
- silverstripe/silverstripe-cms (finished)
- silverstripe/silverstripe-admin (errored half way through)
- silverstripe/silverstripe-versioned (never started)
- silverstripe/silverstripe-assets (never started)
It would be useful to mark "startingrepo = silverstripe/silverstripe-admin" in the bookmark data, and on that basis rotate use the following list order for my next run:
- silverstripe/silverstripe-admin
- silverstripe/silverstripe-versioned
- silverstripe/silverstripe-assets
- silverstripe/silverstripe-framework
- silverstripe/silverstripe-cms
You can see that the items prior to silverstripe-admin have been cut from the top of the list and appended to the bottom.
What do you think? Would this be a straightforward way of improving reliability for large datasets
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating how bookmark data is stored and how repositories are ordered and fetched. Trace the import flow described in the issue, including the point where a repository errors or remains unstarted. Done means persisting the interrupted repository and rotating the next run's order so it resumes there.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github, python
- Domain
- api, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100