singer-io / singer-io/tap-github

Suggestion: mark "starting repository" in bookmark data

Open
#46 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
76
Forks
86
PR merge metrics
No merged PRs in 30d

Description

If you had a very large number of repositories being fetched, say 50-100, it may be that you can never get through everything in an hour before you hit your 5000 request limit. I haven't hit this issue myself yet (well, I'd want to see where we get to with #43), so it's a little hypothetical, but I think it could become an issue in a future use-case (I'm likely to want to track about 80+ repos in my full implementation)

In such as a case, it would be absolutely fine to try again with the next fetch in an hour's time, but I would want to pick up where I left off.

So let's say I was tracking these 3 repos:

  • silverstripe/silverstripe-framework
  • silverstripe/silverstripe-cms
  • silverstripe/silverstripe-admin
  • silverstripe/silverstripe-versioned
  • silverstripe/silverstripe-assets

And in my import I got

  • silverstripe/silverstripe-framework (finished)
  • silverstripe/silverstripe-cms (finished)
  • silverstripe/silverstripe-admin (errored half way through)
  • silverstripe/silverstripe-versioned (never started)
  • silverstripe/silverstripe-assets (never started)

It would be useful to mark "startingrepo = silverstripe/silverstripe-admin" in the bookmark data, and on that basis rotate use the following list order for my next run:

  • silverstripe/silverstripe-admin
  • silverstripe/silverstripe-versioned
  • silverstripe/silverstripe-assets
  • silverstripe/silverstripe-framework
  • silverstripe/silverstripe-cms

You can see that the items prior to silverstripe-admin have been cut from the top of the list and appended to the bottom.

What do you think? Would this be a straightforward way of improving reliability for large datasets

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating how bookmark data is stored and how repositories are ordered and fetched. Trace the import flow described in the issue, including the point where a repository errors or remains unstarted. Done means persisting the interrupted repository and rotating the next run's order so it resumes there.

Written by the indexing model from the issue text.

Assessment

Tech stack
github, python
Domain
api, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.