societe-generale / societe-generale/github-crawler
need throttling options
Nobody has claimed this yet.
- Dominant language
- Kotlin
- Stars
- 111
- Forks
- 21
- PR merge metrics
- No merged PRs in 30d
Description
Summary
in some cases, rate limitation may be in place and pretty restrictive, even for authenticated API requests.
We would need a throttling config param, under which we can specify :
- the throttling period in seconds
- the max number of repositories we can process during the throttling period.
if we cross the limit, we need to wait for the throttling period before we continue.
While we're at it, we can move the existing crawlInParallel property under the throttling property, as it's related.
Type of Issue
It is a :
- request
Your Environment
- Version used: 1.0.11
- OS and version:
- Version of libs used:
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the existing crawlInParallel configuration and the loop that processes repositories, then inspect how API requests are currently handled. Done means supporting a throttling period, a maximum repository count per period, waiting after the limit is crossed, and relocating crawlInParallel under the throttling configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kotlin
- Domain
- api
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100