singer-io / singer-io/tap-github
Pull index of all public repositories using the `repositories` endpoint
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 76
- Forks
- 86
- PR merge metrics
- No merged PRs in 30d
Description
I'd like to pull a list of all public repositories using the repositories endpoint described here: https://docs.github.com/en/rest/reference/repos#list-public-repositories
API info
I've tested the endpoint with some success. The incremental replication key is 'since' - which accepts not a datetime, but an incremental integer identify column which is applied to each repository as it is created. Apparently, this key can be used to paginate through all public repos on github.
Use case
The use case here would be to collect repo ids we could use to then collect follow-up metrics - specifically for repos match a naming conventions for Singer plugins and forks: tap-<something>, target-<something>, pipelinewise-tap-<something>, etc. Once we collect the repo names and IDs we would collect and aggregate additional github metrics on usage, commits, etc.
More info here: https://gitlab.com/meltano/singerhub/-/issues/3 and https://gitlab.com/meltano/singerhub/-/issues/11
As part of our initiative to make taps more discoverable and to help the Singer/Stitch/Meltano community members more quickly locate and evaluate from the large (and growing) list of available taps and targets.
New vs existing tap
I know the paradigm we have here in this tap expects a set of specific repos to extract, and this application would break with that paradigm. If it is preferable to spin this off as a separate tap, I would understand that argument and in that case would likely try to spin off a fork specifically for the purpose of parsing the github index (maybe tap-github-index?).
Expected volume of data
The volume of data is large, but not prohibitively so.
Looks like approximately 48 million public repos according to a quick github search:
https://github.com/search?q=is:public
This is up from 28 million approximately a year ago:

Sample record:
[
{
"id": 1,
"node_id": "MDEwOlJlcG9zaXRvcnkx",
"name": "grit",
"full_name": "mojombo/grit",
"private": false,
"owner": {
"login": "mojombo",
"id": 1,
"node_id": "MDQ6VXNlcjE=",
"avatar_url": "https://avatars.githubusercontent.com/u/1?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/mojombo",
"html_url": "https://github.com/mojombo",
"followers_url": "https://api.github.com/users/mojombo/followers",
"following_url": "https://api.github.com/users/mojombo/following{/other_user}",
"gists_url": "https://api.github.com/users/mojombo/gists{/gist_id}",
"starred_url": "https://api.github.com/users/mojombo/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/mojombo/subscriptions",
"organizations_url": "https://api.github.com/users/mojombo/orgs",
"repos_url": "https://api.github.com/users/mojombo/repos",
"events_url": "https://api.github.com/users/mojombo/events{/privacy}",
"received_events_url": "https://api.github.com/users/mojombo/received_events",
"type": "User",
"site_admin": false
},
//...
]
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the GitHub REST API public repositories endpoint and the existing tap paradigm described in the issue. Determine whether this belongs in tap-github or a separate tap, including how the since pagination and large result volume should be handled. Done requires a decided scope for collecting repository IDs before follow-up metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github, python
- Domain
- api, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100