Fetch files from GitHub using Contents API
- Dominant language
- JavaScript
- Stars
- 27.2k
- Forks
- 5.6k
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 46
Description
Quite a lot of our badges involve fetching the contents of a file from GitHub.
When we do this, we usually do it using https://raw.githubusercontent.com/ instead of the GitHub contents API because this means we can get a file without using a rate limit point.
However
GitHub have recently imposed stricter API rate limits on anonymous access. Relevant reading:
- https://github.blog/changelog/2025-05-08-updated-rate-limits-for-unauthenticated-requests/
- https://github.com/orgs/community/discussions/159123#discussioncomment-13148279
Hilariously, GitHub are getting smashed by LLM scrapers. I wonder if anyone at GitHub has ever heard the phrase "hoisted by their own petard" 😂 I guess not.
In any case, we are starting to see 429s calling raw.githubusercontent.com. I am seeing some of these thrown from production, but not as many as I'd expect. I reckon on GitHub's side we might actually have a slightly higher rate limit but I don't expect anyone to confirm or deny that. It is definitely affecting us both in CI and running tests in local dev, making tests flaky.
Switching to using the GitHub contents API should fix the 429s at the expense of costing us some rate limit points. I'm not too worried about this. We have some tokens to play with. Some places we can just change our code internally. However fixing this across the board is going to be non-trivial as there are a lot of places where we accept a user-supplied URL because we've encouraged users to supply a raw.githubusercontent.com URL. I'm thinking of situations like https://shields.io/badges/oss-lifecycle
To really put this to bed once and for all probably need to implement something at the HTTP client layer to internally call the GitHub contents API when we see a request to https://raw.githubusercontent.com We don't need to boil this whole ocean at once though
Sigh
Contributor guide
Research direction
Start by locating the HTTP client layer that handles requests to raw.githubusercontent.com and the badge paths that accept user-supplied raw URLs. Review the GitHub Contents API and the existing CI and local tests for 429 failures; done means affected requests use the Contents API reliably without breaking supported badge URLs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github, javascript
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100