QuantEcon / QuantEcon/workflow-backups
ENH: Optimize GitHub API usage for issues metadata backup
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Summary
The issues metadata backup feature (added in v0.3.0) works but has performance and rate limit concerns that need addressing before enabling in production.
Current Implementation
- Uses PyGithub REST API
- 1 API call per page of issues (30 issues/page)
- 1 API call per issue to fetch comments
- For a repo with 100 issues → ~100+ API calls
Test Results
Backing up QuantEcon.manual (29 issues):
- Time: ~38 seconds (~1.3s per issue)
- API calls: ~30
- Output: 43 KB JSON file
Concerns for Full Org Backup
- QuantEcon has ~100 active repos
- If average 50 issues per repo = 5,000+ API calls
- GitHub Actions GITHUB_TOKEN limit: 1,000 requests/hour
- Could easily hit rate limits
Proposed Solutions
Option 1: GraphQL API (Recommended)
Use GitHub GraphQL API to fetch issues + comments in a single query per repo.
query {
repository(owner: "QuantEcon", name: "quantecon-py") {
issues(first: 100, states: [OPEN, CLOSED]) {
nodes {
number
title
body
comments(first: 100) {
nodes { author { login } body createdAt }
}
}
}
}
}
Benefits:
- Single request per repo (with pagination)
- Dramatically fewer API calls
- Faster execution
Option 2: Add include_comments config option
backup_metadata:
issues: true
include_comments: false # Skip comments, much faster
Option 3: Rate limit handling
Add retry logic with exponential backoff when rate limited.
Current Status
- Feature implemented and tested ✅
- Default disabled (
issues: false) until optimized - Config includes warning comment about API usage
Related
- Issues backup JSON schema is finalized and working
- Markdown recovery utility planned for future
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the existing issues metadata backup implementation and its PyGithub calls, then compare the proposed GraphQL, comment-skipping, and retry approaches. Done should mean substantially fewer API requests for repository backups while preserving the finalized issues backup JSON schema and handling GitHub rate limits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python
- Domain
- api, devops
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100